Idempotency
An operation is idempotent when doing it again changes nothing further. It is a property of the effect, not of the request: a retry is safe because the second attempt lands on a system that has already recorded the first. There are three ways to get there — design the operation so repetition is meaningless, carry a key that identifies the attempt, or let a uniqueness constraint decide.
Why this is not optional#
Every distributed system has one unavoidable ambiguity: a request that times out may have succeeded. The caller cannot tell a lost request from a lost reply, and no amount of instrumentation removes that — the failure happens in the gap between the two. Retrying is therefore mandatory, and so is making the retry harmless.
The stakes rise sharply when the effect leaves your system. A duplicated read costs a query. A duplicated send is a customer receiving the same message twice; a duplicated call is a customer's phone ringing twice, which is worse, because there is no version of that they experience as a technical detail.
Three mechanisms#
| Mechanism | How it works | Best for |
|---|---|---|
| Natural idempotence | Express the operation as *set to this value* rather than *change by this amount* | State updates: marking read, setting a stage, closing a case |
| An idempotency key | The caller invents a key per attempt; the server stores it with the outcome and replays that outcome for any repeat | Anything that creates a record or triggers an outbound action |
| A uniqueness constraint | A natural key — a provider event id, a follow-up plus its due date — is unique in the database, and a duplicate insert simply loses | Receiving events you did not originate |
The third deserves its own note because the obvious alternative is wrong. Checking whether you have seen something and then writing it is two operations with a race between them; two concurrent copies both read *no* and both proceed. A constraint is decided by the database in one step, and catching the conflict is the design rather than an error path.
Does Connect use idempotency?#
Used, and visible in behaviour you can observe. The clearest example is the outbound boundary: a send is not recorded as *sent* until the provider's own acknowledgement arrives. Until then the state is *uncertain*, and treating uncertain as sent — or as failed — is exactly the mistake that produces a duplicate or a silence.
The phone follow-up drain shows the same discipline with a time bound. runner.chase_phone drains phone follow-ups two per tick through the same gates as any outbound call, and records each outcome explicitly: call placed with an identifier, awaiting approval with an identifier, or refused with a reason. A line that is not ready is retried in an hour. A row more than 24 hours late is closed as missed and never rung — because a very late call-back is worse than none, and quietly retrying forever is how that happens.
Inbound events take the two-stage route: what a provider sent lands in the provider-shaped tenant_* tables, and the canonical threads, messages and contacts are bridged from them. A repeated delivery adds a copy of an arrival, not a second conversation.
The actions where it cannot save you#
- Anything already delivered. Once a message reaches a provider, no key retracts it. Idempotency prevents a second send; it does not undo the first.
- Effects a person perceives. A duplicate that only touches your database is an internal problem. A duplicate that rings a phone is a relationship problem, and it is the reason the outbound gates sit before the action rather than after it.
- Distinct attempts that look alike. Two genuine follow-ups to the same person on the same day are not duplicates, and a deduplication rule crude enough to merge them has traded one failure for a quieter one.
- Non-deterministic content. Replaying a stored outcome is safe. Regenerating a reply and sending 'the same' message is not — the second one will not be the same, and the recipient will notice.
Questions#
Is a GET automatically idempotent?
It is supposed to be, and mostly is. Where it is not is when a read has a side effect somebody added later — a counter, a last-seen timestamp, a lazily created record. Those are worth finding, because caches and prefetchers repeat reads without asking anyone.
How long should an idempotency key be honoured?
Long enough to cover every retry a sane client will make, and no longer — a window measured in hours rather than for ever. Storing them permanently turns a safety mechanism into a table nobody prunes.
If a send is uncertain, should the system retry or stop?
Stop, surface it, and let the record say uncertain. That is the whole reason the state exists: an automatic retry on an uncertain send is a coin flip between a duplicate and a delivery, and a person deciding with the evidence in front of them beats both.