Idempotency Keys for Money-Safe Endpoints
It is 11pm on launch day. Support pings you: two customers say their card was charged twice for the same order. You open the logs and there it is, plain as day. Two identical POST /charges, same customer, same amount, 900 milliseconds apart. Two charge ids. Two receipts. One tweet already being drafted.
Here is how it happened. The Pay button had no disabled state, the customer was on hotel wifi, and the first response was slow. So they tapped again. The browser fired the second request before the first one came back. Both requests were well formed, both reached your handler, and your handler did exactly what you told it to. It charged the card. Twice.
None of this is exotic. A double tap does it. A flaky network and a client that retries on timeout does it. A load balancer that re-sends an idle request does it. A queue that redelivers a message does it. The fix is not to pray the second request never arrives, because it will. The fix is to make the second request harmless.
Safe and idempotent are not the same word
Two words get thrown around here and people blur them. They mean different things.
A method is safe when it only reads. It has no intended effect on the server at all. GET and HEAD are safe. You can fire a hundred of them and nothing on the server changes.
A method is idempotent when making the same request many times leaves the server in the same state as making it once. The state is what matters, not the response you get back. Safe methods are automatically idempotent (reading changes nothing), but idempotent is the wider circle: it also covers methods that do change state, as long as repeating the change is a no-op.
Why is PUT /users/42 idempotent but POST /charges is not? Because PUT names the exact thing it is writing. “Set user 42 to this object.” Run it once or ten times, user 42 ends up the same. DELETE /users/42 is the same idea: after the first call the user is gone, and calling it again does not un-delete or double-delete anything. The server state is identical.
POST is the odd one out. It means “process this, and create whatever that implies.” Each call is a fresh act of creation. Two POST /charges mean two charges, by design. That is not a bug in POST. It is what POST is for. Which is exactly why POST is the method that needs help, and the help has a name.
The pattern: a key the client owns
The idea is small. The client attaches a unique idempotency key to the request, usually in an Idempotency-Key header. The key stands for “this one logical operation.” The server keeps a record of every key it has seen along with the response it produced. When a request shows up with a key the server already finished, the server skips the work and replays the stored response.
On the client, generate the key once per logical operation and reuse it for every retry of that same operation. crypto.randomUUID() is the easy choice. The rule that trips people up: a fresh key per retry defeats the entire mechanism, because each retry then looks like a brand new payment.
const key = crypto.randomUUID(); // one key for THIS payment, reused on retry
async function pay(order) {
return fetch("/charges", {
method: "POST",
headers: {
"Content-Type": "application/json",
"Idempotency-Key": key,
},
body: JSON.stringify(order),
});
}The header is just a string the client picked. Everything interesting happens on the server, and the interesting part is not “look the key up and return the saved response.” Anyone can write that. The interesting part is the three or four ways that naive version is still broken.
See it fail and then hold. The button below fires two requests the way a double tap does. Flip the toggle and watch the charge counter go to 2, or stay at 1:
Store the key with the effect, not next to it
Here is the version almost everyone writes first. It has two bugs stacked on top of each other.
// Do NOT ship this. Two separate bugs live here.
app.post("/charges", async (req, res) => {
const key = req.get("Idempotency-Key");
const seen = await db.query(
"SELECT response FROM idempotency_keys WHERE id = $1", [key]);
if (seen.rows.length) return res.json(seen.rows[0].response);
const charge = await chargeCard(req.body); // the effect
await db.query(
"INSERT INTO idempotency_keys (id, response) VALUES ($1, $2)",
[key, charge]); // written afterwards
res.json(charge);
});Start with the second bug, because it is the one that actually charges people twice. The card gets charged on one line, and the key gets written on a later line, in a separate statement. Those two writes are not atomic. If the process crashes, or the database connection drops, or the pod gets evicted in the gap between them, you have moved money and recorded nothing. The retry finds no key, so it charges again.
The whole value of an idempotency key comes from one rule:
Write the key in the same transaction as the effect. They commit together or not at all.
There is an honest wrinkle here, and skipping it would be lying to you. If the effect lives in another system, a real payment processor, you cannot pull that charge into your database transaction. A rollback in Postgres does not un-charge a Visa card. The answer is to layer keys: send your own idempotency key downstream to the processor so the actual money movement dedupes at the source, and store the processor’s result in your transaction. Your key protects your endpoint, their key protects the money. For an effect that is purely local (moving an internal balance, writing a row), one COMMIT is the whole story.
Let the database win the race
Back to the first bug, the SELECT then INSERT. It reads fine until you picture two requests arriving at the same millisecond, which is precisely the double-tap scenario you are defending against.
Request A runs the SELECT, sees no key, and starts charging. Request B runs its SELECT a hair later, also sees no key (A has not written it yet), and also starts charging. Both pass the check. Both charge. The check-then-act pattern has a hole exactly the width of the time between the two statements, and concurrency will drive a truck through it. This is a classic time-of-check to time-of-use race.
You cannot close that hole by checking more carefully. You close it by not checking at all. Make the key column carry a UNIQUE constraint and let the database be the referee. Both requests try to INSERT the same key. The database guarantees exactly one of them wins. The other gets a duplicate-key error, which is not a failure, it is your signal.
Postgres raises unique_violation with SQLSTATE 23505 on a duplicate. You can catch that code, or you can say it declaratively with ON CONFLICT. Here is the whole handler, claiming the key and doing the work in one transaction:
app.post("/charges", async (req, res) => {
const key = req.get("Idempotency-Key");
if (!key) return res.status(400).json({ error: "Idempotency-Key required" });
const fingerprint = sha256(canonicalize(req.body));
const client = await pool.connect();
try {
await client.query("BEGIN");
// Claim the key. On a duplicate, ON CONFLICT DO NOTHING
// returns zero rows, which is how the loser learns it lost.
const claim = await client.query(
`INSERT INTO idempotency_keys (user_id, key, request_hash, status)
VALUES ($1, $2, $3, 'in_progress')
ON CONFLICT (user_id, key) DO NOTHING
RETURNING status`,
[req.userId, key, fingerprint]);
if (claim.rows.length === 0) {
await client.query("ROLLBACK");
return replayOrReject(res, req.userId, key, fingerprint);
}
const charge = await chargeCard(req.body); // the effect
await client.query(
`UPDATE idempotency_keys
SET status = 'succeeded', response = $3
WHERE user_id = $1 AND key = $2`,
[req.userId, key, charge]);
await client.query("COMMIT");
res.status(201).json(charge);
} catch (err) {
await client.query("ROLLBACK");
throw err;
} finally {
client.release();
}
});Notice what ROLLBACK buys you in the catch. If chargeCard throws before the commit, the claim row you inserted this transaction disappears with it. The key becomes unclaimed again, so a genuine retry is free to try once more. A failure that never committed leaves no trace, which is exactly what you want.
The duplicate that arrives mid-flight
The race diagram had a tidy ending: the loser reads the winner’s response and replays it. But look at the timing. The loser hits the conflict while the winner is still charging the card. There is no response to replay yet. The status is still in_progress.
This is the case the naive version never even considers, and it is common. Two taps, back to back, the first one still talking to the payment processor when the second arrives. You have two sane options.
The simple option is to reject the mid-flight duplicate with 409 Conflict and let the client try again in a moment. By then the first request has usually committed, and the retry becomes a clean replay. The other option is to make the second request wait for the first to finish and then hand back its result. Waiting is a nicer experience but more machinery, since you need a way to block on the in-flight request without pinning a connection forever.
Either way you need a locked_at timestamp and a timeout. If a request claims a key, marks it in_progress, and then the process dies, that key is stuck. Nothing will ever move it to succeeded. So a duplicate that finds an in_progress row whose locked_at is older than your timeout (30 to 60 seconds is typical) should be allowed to assume the original died and take over. Without that escape hatch, one crashed request wedges a customer’s key permanently.
async function replayOrReject(res, userId, key, fingerprint) {
const { rows } = await pool.query(
`SELECT status, response, request_hash, locked_at
FROM idempotency_keys WHERE user_id = $1 AND key = $2`,
[userId, key]);
const row = rows[0];
if (!row) return res.status(409).json({ error: "retry" }); // pruned mid-race
if (!row.request_hash.equals(fingerprint)) {
return res.status(422).json({ error: "key reused with a different body" });
}
if (row.status === "in_progress") {
return res.status(409).json({ error: "original request in flight" });
}
return res.status(200).json(row.response); // the real replay
}Scope and lifetime
Two more decisions turn a demo into something you would run.
Scope the key. A raw key column invites collisions across tenants: two customers both generate a UUID, and in the astronomically unlikely (but not impossible, especially with weak client key generation) event they match, one reads the other’s charge. Scope the uniqueness to the account. A composite key does both jobs at once, uniqueness and isolation:
CREATE TABLE idempotency_keys (
user_id bigint NOT NULL,
key text NOT NULL,
request_hash bytea NOT NULL,
status text NOT NULL DEFAULT 'in_progress',
response jsonb,
locked_at timestamptz NOT NULL DEFAULT now(),
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (user_id, key)
);Give keys a lifetime. You are not keeping these forever. Most implementations prune keys after roughly 24 hours, which is the window Stripe uses too. Long enough to cover any sane retry loop, short enough that the table does not grow without bound. After a key is pruned, it is treated as new, so a retry that shows up two days late will run again. That is fine, because no reasonable client retries a payment a day later.
The request_hash column earns its place here. Store a hash of the (canonicalized) request body when you first see a key. When a key comes back, compare the incoming body’s hash against the stored one. Same hash, replay. Different hash, that is the 422: the client reused a key for a different operation, and blindly replaying the old response would be worse than an error. Canonicalize before hashing (sorted keys, stable formatting) or two semantically identical bodies will hash differently and trip a false 422. Input hashing pairs naturally with validating input.
When you do not need a key at all
An idempotency key is a general tool for making an arbitrary POST safe to retry. Sometimes you can sidestep the whole key store because the operation is naturally idempotent, or can be made so. When that is available, it is usually simpler and worth reaching for first.
The cleanest version: derive a deterministic id from stable inputs and let a unique constraint dedupe. If the client supplies an order_id, or you hash the meaningful fields into one, then two requests for the same order carry the same id, and the second insert is a harmless no-op.
-- Deterministic id from the client. The DB dedupes for free.
INSERT INTO orders (id, user_id, amount)
VALUES ($1, $2, $3)
ON CONFLICT (id) DO NOTHING;The same instinct shows up as a conditional update, sometimes called compare-and-set. Instead of “mark this invoice paid,” say “mark this invoice paid only if it is currently pending.” The first request matches and updates one row. Every repeat matches zero rows and changes nothing.
UPDATE invoices SET status = 'paid', paid_at = now()
WHERE id = $1 AND status = 'pending';
-- affected rows: 1 the first time, 0 on every retryAnd an upsert with ON CONFLICT DO UPDATE handles “create or overwrite” idempotently, since setting a row to a fixed value twice lands on the same value. The through-line for all three: pin the operation to a specific target state or a specific id, so repeating it is definitionally a no-op. No side table, no key expiry, no lock. Reach for a full idempotency key store when the operation genuinely cannot be expressed this way, which is often the case once an external charge is involved.
The same problem wears other hats
Once the shape clicks, you start seeing it everywhere a message can be delivered more than once.
Webhooks. A provider that sends you events will retry them on any hiccup, and “did the customer’s request time out” is a hiccup. So the same charge.succeeded lands twice. Every serious provider stamps each event with a stable id precisely so you can dedupe. Record the event id, and on a repeat, acknowledge and skip. That is an idempotency key by another name, and it is covered from the receiving side in webhooks.
Queue consumers. Almost every queue worth using delivers at least once, not exactly once. Redelivery after a consumer crash, a visibility timeout that fired early, a rebalance: all of them hand you the same message twice. A consumer that increments a balance or sends an email has to dedupe on the message id, or it double-counts. Exactly-once delivery is mostly a marketing phrase. Idempotent consumers are how you get exactly-once effects out of at-least-once delivery.
Plain retries. The subtle one. A client fires a request, the network drops the response on the way back, and the client sees a timeout. Did the write happen? The client has no idea. The server may have committed and the ack got lost. So the client retries, and without a key that retry is a second write. This is why timeout does not mean failure, and why a retry policy without idempotency is a slow-motion double-charge machine. This project’s own checkout, the button that sells the course PDF, is exactly this shape: a single POST where a lost response must be safe to retry.
Where this stands
The pattern is evergreen and the mechanics are not going anywhere. There is an Idempotency-Key header convention that the industry converged on, popularized by Stripe and echoed by most payment and infrastructure APIs. There is also an IETF Internet-Draft in the httpapi working group, draft-ietf-httpapi-idempotency-key-header, sitting at revision 07 as of late 2025. Be honest with yourself about its status: it is a draft, it has lapsed and been revived more than once, and it is not a finished RFC. Treat it as a well-reasoned description of the convention, not a standard you can cite as settled. The 409 and 422 semantics above come from it, and they are sensible whether or not it ever ships.
The important parts do not depend on any spec. Store the key with the effect, in one transaction. Let a unique constraint arbitrate the race. Handle the in-flight duplicate on purpose. Scope keys and expire them. Prefer natural idempotency when the operation allows it. Do those five things and the tap-Pay-twice bug simply cannot happen, no matter how the second request gets there.
Summary
- Safe means read-only; idempotent means repeating the request leaves the same server state.
GET,HEAD,PUT, andDELETEare idempotent;POSTandPATCHare not, which is why POST endpoints need protection. - The client sends an
Idempotency-Key(a UUID, generated once and reused on every retry of the same operation), and the server maps key to stored response so a repeat replays instead of re-running. - Store the key row in the same transaction as the effect. Separate writes leave a crash gap that re-charges. For an external effect you cannot roll back, forward an idempotency key downstream and store its result locally.
- Let the database win the race. A
SELECTthenINSERThas a time-of-check to time-of-use hole; aUNIQUE (user_id, key)constraint withON CONFLICT DO NOTHING RETURNINGpicks one winner atomically. - Handle the mid-flight duplicate. Track
in_progressvssucceeded, return409(or wait) while the first runs, and use alocked_attimeout so a crashed request does not wedge the key. - Scope keys to the account, expire them (about 24h), and hash the request body to reject a reused key that carries a different payload with
422. - Prefer natural idempotency where you can: deterministic ids with
ON CONFLICT DO NOTHING, conditional (compare-and-set) updates, and upserts need no key store at all. - The same dedupe shows up in webhooks, at-least-once queue consumers, and any retry after a lost response. At-least-once delivery plus idempotent effects is how you actually get exactly-once.