Circuit breakers
A circuit breaker is a piece of state in front of a remote dependency. It counts recent failures, and once there are enough it stops calling the dependency at all for a while, answering immediately instead. The point is not to save the provider. It is to stop your own workers queueing behind calls that were going to fail.
Three states and the one that does the work#
The pattern has three states, and almost every discussion of it concentrates on the wrong two. Closed and open are easy. Half-open is where the design decisions live.
- Closed
- Calls pass through. Failures are counted against a window — a rolling count, a rate, or both. Nothing is different from having no breaker at all, except the bookkeeping.
- Open
- The threshold was crossed. Calls are refused locally without touching the network, and the caller gets a refusal in microseconds rather than a timeout in tens of seconds.
- Half-open
- The cool-down expired. A small number of calls — often exactly one — are let through as probes. Success closes the breaker; failure re-opens it and usually lengthens the next cool-down.
The half-open state is what separates a breaker from a kill switch. A kill switch needs a person to turn it back on; a breaker recovers by itself, and the cost of that recovery is bounded to however many probe calls you allow. Allow too many and the recovery attempt is itself a thundering herd against a service that has just come back.
What it protects, and what it does not#
A breaker protects the caller. That is worth stating plainly, because the pattern is often introduced as politeness towards an overloaded dependency, and then judged by whether the dependency recovered faster. The measurable benefit is on your side of the wire: threads, connections, memory and queue depth that would otherwise be consumed by requests with no chance of succeeding.
| Failure | Breaker helps? | Why |
|---|---|---|
| Provider is down; connections refused instantly | Barely | The call already fails fast. There is little resource to reclaim. |
| Provider accepts connections and never answers | Strongly | This is the case that eats every worker. Timeouts are long by comparison with everything else. |
| Provider is rate-limiting with 429s | Partly | A rate limiter or a retry budget fits better; the correct response is to slow down, not to stop. |
| One bad request fails repeatedly | No | Nothing is wrong with the dependency. Retrying a request that will never succeed is a caller bug. |
Does Connect use a circuit breaker?#
Educational, with the same effect achieved differently. The Connect capability registry lists no circuit-breaker component, and no page here should be read as saying one exists. What the registry does list is readiness and health state, checked before work is attempted rather than counted after it fails — which lands in much the same place for the failures that matter here.
- Line health.
voice_engine.line_healthis a verdict about a phone line rather than a counter of recent errors. A line that is not ready is not rung; the phone follow-up drain recordsrefused: <reason>and retries in an hour. - Mailbox health. A mailbox that authenticates and returns nothing is recorded as a *quiet mailbox*, which is a health signal, not an error. Health verdicts sit on the mailbox row and surface on the Mailboxes screen.
- Readiness for messaging.
readiness.messagingsays on the Phone screen that the configured carrier carries no SMS, rather than accepting an outbound message that would never leave. - Giving up on time rather than on failure count. A phone follow-up more than 24 hours late is closed as
missed:and never rung. The rule is about the value of the work, not the health of the provider.
The one place Connect deliberately does *not* fail fast is mail intake. When the daily allowance is spent, work is held rather than dropped and the read cursor does not advance. A breaker's instinct — refuse and move on — is the wrong instinct for a queue whose position is the only record that a message was ever seen.
If you are building one#
Decide what counts as a failure before you count anything. A 404 from a lookup is usually a valid answer, not a fault.
Result The breaker stops opening on ordinary business outcomes, which is the most common way this pattern is mis-tuned.
Give the open state a useful refusal — the dependency name and when it will next be probed.
Result Callers can distinguish 'refused locally' from 'the provider said no', which are different problems with different owners.
Emit the state transitions, not the state.
Result Two lines an hour when something breaks and recovers, rather than a gauge nobody reads. A breaker that opens and closes repeatedly is a signal in itself.
Questions#
Is a circuit breaker the same as a retry limit?
No, and they solve opposite halves of one problem. A retry limit bounds how hard one request tries; a breaker bounds how many requests are allowed to try at all. Retries without a breaker turn a slow dependency into an outage, because every caller multiplies its own load. A breaker without retries throws away work that a second attempt would have completed.
What should a caller do when the breaker is open?
Whatever it would do for a refusal it can explain, which is rarely an error page. In Connect's equivalents the answer is to hold the work and say so: a held reply stays in Needs You, a follow-up is retried later, a refused call is recorded with its reason. Discarding the work is the one option that cannot be undone.
Does an open breaker mean the provider is down?
It means calls from this process to that dependency have been failing. That is not the same thing — a bad credential, a DNS problem or a firewall change on your side produces an identical pattern. Treat an open breaker as a question, and confirm the provider's own status before telling anyone it is the provider's fault.