Idempotency for retried telephony webhooks
A carrier cannot tell a lost response from a slow one, so it retries, and every telephony webhook is delivered at least once and sometimes more. Connect stores each normalised provider event exactly once in call_events, with the signature nonce under a partial unique index; the second delivery collides and is discarded. What is retried outbound is then decided by what a request means, not by whether it failed.
Why the same event arrives twice#
A carrier posts an event and waits. If the answer is slow, lost in a proxy, or arrives after a deploy has restarted the process, the carrier has no way to know whether the event was handled. Its only safe move is to send it again. Any telephony integration therefore receives duplicates as normal traffic rather than as an anomaly.
The damage a duplicate does depends on what the handler treats as new. A second *answered* event can restart a turn; a second *completed* can close a call twice and charge it twice; a replayed event with a valid signature is a replay attack rather than an accident. All three are the same shape of problem: the handler acting on a delivery rather than on an event.
One row per event, enforced by the database#
call_events stores every normalised provider event once. The carrier's signature nonce sits under a partial unique index, so the second arrival of the same signed event violates the constraint and is discarded before it reaches any logic. The protection is in the schema rather than in a remembered set in a process, which matters because a process restart forgets and a database does not.
Normalising first is what makes one index enough. The adapter (voice_carriers.py) is the only place a provider's own words appear; by the time an event reaches call_events it has a shape that does not depend on which carrier sent it, so a second carrier does not need a second deduplication scheme.
A duplicate is not the only thing a handler must read#
Deduplication answers *have I seen this exact delivery*. It does not answer *should this delivery do anything*. /voice/{carrier}/status was both the hangup URL and the status URL, and it ended the call on any status — so a ringing callback, entirely legitimate and not a duplicate at all, hung up on a live caller. Reading the status before acting on it is a separate discipline from replay protection, and skipping it fails in a way that sounds exactly like a network fault.
The same lesson appears in Direction, which was never read. A carrier posts to the *same* answer URL when a customer calls in and when one of the business's own SIP endpoints calls out. An agent dialling from the browser was greeted as though they were the customer, and the person they dialled never rang.
What Connect retries, and what it treats as an answer#
| Request or failure | Rule |
|---|---|
arrived, brief, end | A call cannot lose these: retried through a deploy restart or a proxy 502 |
| Any 4xx | An answer, never retried — the far side has decided |
| Dial answered 408 / 503 / 504 | Dialled once more; a timeout is not a decision |
| Leg answered at the carrier that never joins the room | Written off after DIAL_JOIN_TIMEOUT_S instead of waited for |
| Model session refuses to start | Reopened once on the same model, once on the fallback, then given up — model_session_restarted goes on the record |
The split is deliberate: a transport failure is worth repeating, a considered refusal is not. Retrying a 4xx is how an integration turns one rejected request into a sustained one.
The audit that catches what the code cannot#
Reading the carrier's own call log beside the calls table is the fastest check available, and a Completed on their side with no row on yours is a bug every time. It found the worst of these: carrier_incoming read the voice settings before entering the workspace, saw a switched-on line as disabled, answered with the goodbye message and created no Call at all. The carrier logged a completed, charged six-second call that Connect had no record of.
- Idempotency protects the record, not the caller — a discarded duplicate still cost a request and still may have raced a live turn.
- The nonce belongs to the carrier. A provider that does not sign, or signs without a nonce, needs a different key.
- No figure exists here for how often duplicates arrive in production: UNKNOWN.
Questions#
Why a database constraint rather than a cache of recent event ids?
Because the duplicate usually arrives around a restart, which is exactly when an in-process cache is empty. A unique index survives the restart, survives a second worker, and fails the insert rather than relying on a check-then-write that two requests can both pass.
Does a discarded duplicate get logged?
The collision is the signal, and the event that was already stored remains the one on the call. What matters operationally is that the call's timeline holds one of each event, which is what makes a later reconstruction of the call trustworthy.
How should a client of Connect's own API handle the same problem?
The same way round: send an idempotency key, treat a 4xx as an answer, and retry only transport failures. Idempotency in the API and Designing a resilient client cover the outbound direction.