Idempotency keys and safe retries

The retry is not hypothetical. A phone loses signal after the request left and before the response came back; a load balancer times out at thirty seconds while the write takes thirty-one; a queue consumer crashes between doing the work and acknowledging it. In every one of those cases — the network failures the distributed-systems path exists to work through — the client has no way to know whether the work happened, and the only safe thing it can do is ask again.

What the key has to promise#

The client generates the key, once, before the first attempt, and reuses it for every retry of that intention. That is the part teams get wrong: a key generated per HTTP call is not an idempotency key, it is a request id, and it protects nothing.

The server stores the key together with a hash of the request body. If the same key arrives with a different body, that is not a retry — it is a bug or an attack, and the honest response is a 422. The two obvious alternatives are both worse: replaying the first request’s result answers a question nobody asked, and doing the new work under the old key discards the protection the key was there to provide.

The window matters too. Keys cannot be kept forever, and a client retrying after your retention window has passed gets the work done a second time. Twenty-four hours is a common answer; whatever you choose, it belongs in the documentation, because it is part of the contract.

Where the record has to live#

In the same database, in the same transaction, as the effect it is protecting. Not in Redis beside the database, not in a middleware that writes after the handler returns — because every one of those introduces a moment where the effect exists and the key does not, and that moment is exactly the one the retry lands in. It is a transaction-boundary question before it is an idempotency question: state and the record of that state change commit together, or neither does. That ordering is why the architecture path reaches boundaries before it reaches this article.

-- the key and the effect commit together, or not at all
BEGIN;

-- RETURNING yields a row only if THIS statement did the insert.
-- One row: this request owns the work. No row: someone else does.
INSERT INTO idempotency (key, request_hash, status)
VALUES ('k_9f2c', '8ab1…', 'in_flight')
ON CONFLICT (key) DO NOTHING
RETURNING key;

-- Only the request that won the claim runs the next two statements.
-- (Not one statement with WITH: in Postgres a data-modifying CTE and the
-- main query share one snapshot, so an UPDATE in the same statement would
-- never see the row the INSERT just claimed.)
INSERT INTO orders (customer, item_id) VALUES (4711, 88);

UPDATE idempotency SET status = 'succeeded', response = '{…}'
  WHERE key = 'k_9f2c';

COMMIT;

-- a replay claims nothing, orders nothing, and reads the stored result
SELECT status, response FROM idempotency WHERE key = 'k_9f2c';

The empty RETURNING is the whole lock: it is how a request learns it lost the race. Whichever one inserted the row owns the work, and only that request runs the statements below — so a replay reaches the SELECT having written nothing at all, and returns what the winner stored. Keep the claim and the follow-up as separate statements in the one transaction: folded into a single statement with WITH, the final UPDATE cannot see the row the claim inserted, and the key stays in flight forever.

The four ways it still breaks#

What to return on a replay#

The same status code and the same body as the first time, byte for byte, because the client is entitled to assume it is seeing the original outcome. Adding a header that says the response was replayed is useful for debugging and harmless to clients that ignore it.

A key makes a retry safe. It does not make the retry someone else’s problem, which is the usual reason a queue gets added — and most of the time the queue is the wrong answer.