Retries and backoff
Retry only what is safe to repeat, and only when the failure suggests the request was never answered. Most 4xx responses are answers — a second identical request gets an identical refusal and adds load for nothing. Timeouts, connection failures and 5xx are worth another attempt, spaced by growing delays with randomness in them, under a deadline rather than a count.
Sort the failure before deciding anything#
| Failure | Answered? | Do |
|---|---|---|
| Connection refused, DNS failure | No — nothing was reached | Retry with backoff |
| Timeout | Unknowable — this is the hard case | Retry only if the operation is idempotent; otherwise reconcile before acting |
| 500, 502, 503, 504 | No — the server admits it failed | Retry with backoff, honouring Retry-After if present |
| 400, 422 — malformed or invalid | Yes | Fix the request. Repeating it is noise |
| 401 — not authenticated | Yes | Get credentials once, then retry once. A loop here locks accounts |
| 403 — not permitted | Yes | Stop. On a customer session this is usually the allowlist, and no amount of retrying changes it |
| 404 — nothing there | Yes | Stop. Under row-level security this may be the correct answer rather than a fault |
| 409 — conflict | Yes, informatively | Re-read the current state and decide again; do not resend blindly |
| 429 — too many | Yes, with instructions | Slow down. See rate limiting — this needs a queue change, not a sleep |
The timeout row is the only genuinely difficult one, and it is difficult for a structural reason: the caller cannot distinguish a request that never arrived from a reply that never came back. That is why idempotency is a prerequisite for retrying rather than a companion to it.
Backoff, and the herd it prevents#
Retrying immediately is the worst possible response to an overloaded dependency: it adds load precisely when the service has least capacity to absorb it. Exponential backoff — roughly doubling the wait each time — gives the far end room to recover.
Exponential backoff on its own is still not enough, and the reason is worth stating carefully. If every client fails at the same moment and every client doubles the same delay, they all come back at the same moment. The outage recovers into a wall of synchronised retries and falls over again. Randomising each delay across the whole interval spreads them, and it is the difference between a recovery and a second incident.
attempt 1 -> wait a random value in [0, 1s)
attempt 2 -> wait a random value in [0, 2s)
attempt 3 -> wait a random value in [0, 4s)
attempt 4 -> wait a random value in [0, 8s)
... bounded by a maximum delay, and stopped by a deadline- Budget by deadline, not by attempt count. "Give up after 30 seconds" is a promise to the caller. "Give up after five tries" is a promise about nothing, because the tries take unknown time.
- Cap the delay. Unbounded doubling produces a retry twenty minutes after anyone stopped caring about the answer.
- Honour what you were told. A
Retry-Afterheader is the server telling you when it will be ready. Ignoring it in favour of your own schedule is a choice to be wrong. - Do not retry inside a retry. Nested layers multiply: three attempts at each of three levels is twenty-seven requests for one logical call. Decide which layer owns retrying and make the others fail fast.
Does Connect use retries?#
Used, with deadlines rather than open-ended attempts. The clearest example is the phone follow-up drain: a line that is not ready is retried in an hour, and a follow-up more than 24 hours late is closed as missed and never rung. That second rule is a deliberate refusal to keep retrying, on the grounds that a very late call-back is worse for the person receiving it than no call at all.
The same shape appears in Needs You. Operational problems — a mailbox that has stopped, a voice line that is unhealthy — surface as entries a person can see, and each drains by itself as its cause clears. That is retrying with a visible failure state attached, which is a different thing from retrying quietly until somebody notices the silence.
When to stop retrying entirely#
- The error is a decision
- A refusal recorded against a rule is an outcome, not a fault. Connect records refusals in the audit trail precisely because a refusal is a decision worth keeping.
- The work has expired
- Some work is only valuable on time. A stale attempt succeeding is not a success.
- The dependency is clearly down
- Past a threshold, stop asking and fail fast. That is what a circuit breaker is for, and it protects your own latency as much as the far end's capacity.
- A person needs to decide
- An uncertain send should surface, not resolve itself. Automatic resolution of an ambiguous outcome is a coin flip in a place where the evidence is available to a human.
Questions#
Should a client retry a 429?
Not as a plain retry. A 429 means your rate of requests is the problem, so repeating at the same rate reproduces it. Reduce concurrency, honour any Retry-After, and treat it as a signal to the queue rather than to the individual request.
Is it ever right to retry a 500 on a non-idempotent action?
Only with an idempotency key, so the second attempt is recognised as the same attempt. Without one you are choosing between losing the work and doing it twice, and for anything that reaches a customer, doing it twice is usually the worse half.
How many attempts is reasonable?
Fewer than instinct suggests. Three attempts inside a short deadline handles transient failures; beyond that you are usually waiting on something that needs a person or a different route. Long retry chains mostly convert a fast failure into a slow one.