A provider quota breaker opening and closing
When a model provider refuses for quota or a rejected key, a breaker opens for that provider. Scheduled work asks first and is skipped; work a person is waiting on — a caller on the line, a reply being drafted in front of you — is attempted anyway. The breaker closes on the first call that gets through, or by itself a quarter of an hour after it opened.
Two refusals worth opening for#
Only two classes of failure open a breaker: the account is out of quota, and the key is rejected. They are grouped because they behave identically from the caller's side — a revoked key fails exactly as often and exactly as permanently as an empty account, and hammering either is no more useful than hammering the other. A timeout or a transient error is not a breaker condition; retrying those is reasonable.
The failure this exists to stop had run for four weeks: a scheduled job asking a model for work every half hour, being refused every half hour, and logging the refusal at a level nobody reads. Nothing was broken loudly enough to notice and nothing was getting done.
Who is asking#
A caller on the line and a four-in-the-morning follow-up drain are both just a model call by the time they reach the engine, and they deserve opposite answers when the account is empty. Nothing in the call itself says which it is, so the scheduler marks its own thread as background on the way in, and the check reads that mark.
| Work | Marked as | With the breaker open |
|---|---|---|
| A caller already on the line | Foreground | Attempted. The provider may still refuse, and then the caller hears the closed-line message |
| A reply being drafted while somebody waits | Foreground | Attempted |
| The scheduled engine tick | Background | Skipped, with the reason and the time remaining |
| The follow-up drain, including phone call-backs | Background | Skipped; the rows stay due and are drained later |
| Prospect research and other batch work | Background | Skipped |
Stage by stage#
- Trigger — a model call comes back refused for quota, or with a rejected key.
- User or external event — usually none. The first refusal is often a scheduled job's, at an hour when nobody is watching.
- Authentication and workspace resolution — the breaker is per provider and process-wide, not per workspace: an account that is empty is empty for everybody using it.
- Ingest or request — the failure is classified. Only quota and authentication open a breaker; everything else is returned to the caller as an ordinary error.
- Canonical record — the open breaker is held in the process and mirrored through settings, so the web process and the voice worker agree about a provider one of them discovered was refusing.
- Reasoning — none. This is deliberately mechanical, because a breaker that reasons about whether to open is a breaker that sometimes does not.
- Knowledge, memory and rules — untouched.
- Autonomy and approval — unchanged. Work that is skipped is not rejected: no approval is spent and no decision is recorded against the item itself.
- Action through a provider — background calls do not reach the provider at all while the breaker is open. Foreground calls do.
- Result — skipped work reports why and how long is left; the counted refusals accumulate against the open breaker.
- Relationship, timeline and memory — nothing is written to a relationship. A skipped drain leaves the follow-up due, not missed.
- Audit, usage and Needs You — open breakers appear in the health summary and beside the budget, so the state is visible rather than only logged.
How it closes#
- A call gets through
- Whatever was wrong is over, so the breaker is dropped immediately and the count of refused attempts is logged once. This is the usual ending.
- A quarter of an hour passes
- The breaker expires when it is next read. The window is chosen so a dead key is not retried in a loop, and an account topped up at five past the hour is serving again well before the next.
- The process restarts
- The mirrored state is loaded back, so a restart does not turn a known-bad provider into a fresh round of refusals.
There is no button that forces a breaker shut, and none is needed: the first successful call is the button. If foreground work is succeeding while background work is still being skipped, the breaker is within its window and will close on its own — that is a normal few minutes, not a fault.
The drain afterwards#
Skipped work is not lost work. Follow-ups stay due and are picked up on a later tick; the engine's next pass finds the same queue it left. The one rule with teeth is lateness: a phone follow-up more than a day past its time is closed as missed rather than rung, because a call-back that arrives a day late is worse than one that never comes.
So a breaker open for minutes costs nothing but minutes. A breaker open for a day costs the call-backs that expired inside it, and those show as missed with a reason rather than disappearing — which is the difference between a queue you can audit and one you have to guess about.
Questions#
Does the breaker stop a call that is already connected?
No. A caller on the line is foreground work and is attempted regardless. If the provider refuses the session anyway, the caller hears the closed-line message in the business's voice rather than silence, and the call is recorded with the reason on it.
Is the breaker per workspace?
No, it is per provider. The account being refused is shared, so opening it only for the workspace that happened to discover the refusal would mean every other workspace rediscovering it one refused call at a time.
Can I see whether a breaker is open?
Yes. Open breakers appear in the health summary with the reason, the number of refused attempts and the time remaining. That summary is what stops this being an invisible state that only shows up as work quietly not happening.