# Idempotency for retried telephony webhooks

A carrier cannot tell a lost response from a slow one, so it retries, and every telephony webhook is delivered at least once and sometimes more. Connect stores each normalised provider event exactly once in `call_events`, with the signature nonce under a partial unique index; the second delivery collides and is discarded. What is retried outbound is then decided by what a request means, not by whether it failed.

- **Status:** Available
- **Audience:** developer
- **Channels:** phone
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/research/idempotent-telephony-webhooks/

## Why the same event arrives twice

A carrier posts an event and waits. If the answer is slow, lost in a proxy, or arrives after a deploy has restarted the process, the carrier has no way to know whether the event was handled. Its only safe move is to send it again. Any telephony integration therefore receives duplicates as normal traffic rather than as an anomaly.

The damage a duplicate does depends on what the handler treats as new. A second *answered* event can restart a turn; a second *completed* can close a call twice and charge it twice; a replayed event with a valid signature is a replay attack rather than an accident. All three are the same shape of problem: the handler acting on a delivery rather than on an event.

## One row per event, enforced by the database

`call_events` stores every normalised provider event once. The carrier's signature nonce sits under a **partial unique index**, so the second arrival of the same signed event violates the constraint and is discarded before it reaches any logic. The protection is in the schema rather than in a remembered set in a process, which matters because a process restart forgets and a database does not.

Normalising first is what makes one index enough. The adapter (`voice_carriers.py`) is the only place a provider's own words appear; by the time an event reaches `call_events` it has a shape that does not depend on which carrier sent it, so a second carrier does not need a second deduplication scheme.

> **Note** The index is partial because it only constrains events that carry a nonce. An event without one is not deduplicated this way, and a handler that must be safe for those has to be idempotent by its own logic.

## A duplicate is not the only thing a handler must read

Deduplication answers *have I seen this exact delivery*. It does not answer *should this delivery do anything*. `/voice/{carrier}/status` was both the hangup URL and the status URL, and it ended the call on **any** status — so a `ringing` callback, entirely legitimate and not a duplicate at all, hung up on a live caller. Reading the status before acting on it is a separate discipline from replay protection, and skipping it fails in a way that sounds exactly like a network fault.

The same lesson appears in `Direction`, which was never read. A carrier posts to the *same* answer URL when a customer calls in and when one of the business's own SIP endpoints calls out. An agent dialling from the browser was greeted as though they were the customer, and the person they dialled never rang.

## What Connect retries, and what it treats as an answer

| Request or failure | Rule |
|---|---|
| `arrived`, `brief`, `end` | A call cannot lose these: retried through a deploy restart or a proxy 502 |
| Any 4xx | **An answer, never retried** — the far side has decided |
| Dial answered 408 / 503 / 504 | Dialled once more; a timeout is not a decision |
| Leg answered at the carrier that never joins the room | Written off after `DIAL_JOIN_TIMEOUT_S` instead of waited for |
| Model session refuses to start | Reopened once on the same model, once on the fallback, then given up — `model_session_restarted` goes on the record |

The split is deliberate: a transport failure is worth repeating, a considered refusal is not. Retrying a 4xx is how an integration turns one rejected request into a sustained one.

## The audit that catches what the code cannot

Reading the carrier's own call log beside the `calls` table is the fastest check available, and a `Completed` on their side with no row on yours is a bug every time. It found the worst of these: `carrier_incoming` read the voice settings **before entering the workspace**, saw a switched-on line as disabled, answered with the goodbye message and created no `Call` at all. The carrier logged a completed, charged six-second call that Connect had no record of.

- Idempotency protects the record, not the caller — a discarded duplicate still cost a request and still may have raced a live turn.
- The nonce belongs to the carrier. A provider that does not sign, or signs without a nonce, needs a different key.
- No figure exists here for how often duplicates arrive in production: UNKNOWN.

## Questions

### Why a database constraint rather than a cache of recent event ids?

Because the duplicate usually arrives around a restart, which is exactly when an in-process cache is empty. A unique index survives the restart, survives a second worker, and fails the insert rather than relying on a check-then-write that two requests can both pass.

### Does a discarded duplicate get logged?

The collision is the signal, and the event that was already stored remains the one on the call. What matters operationally is that the call's timeline holds one of each event, which is what makes a later reconstruction of the call trustworthy.

### How should a client of Connect's own API handle the same problem?

The same way round: send an idempotency key, treat a 4xx as an answer, and retry only transport failures. [Idempotency in the API](/developers/api-idempotency/) and [Designing a resilient client](/developers/errors-and-retries/) cover the outbound direction.

## Related

- [Idempotency in the API](https://connectbyjbrh.com/developers/api-idempotency/)
- [Designing telephony that does not name its provider](https://connectbyjbrh.com/research/provider-independent-telephony/)
- [When a call is answered in silence](https://connectbyjbrh.com/research/silent-failure/)
- [Phone and voice in Connect](https://connectbyjbrh.com/docs/phone/)
- [Verifying a Connect webhook signature](https://connectbyjbrh.com/developers/webhook-signatures/)

## What this page is based on

- `docs-source/sources/PHONE.md` §1 — `channel_routes`, `calls` and `call_events`
- `docs-source/sources/PHONE.md` §2 — six production defects found on the live endpoint
- `docs-source/sources/PHONE.md` §9 — retry, refusal and attribution rules