Webhooks: Receiving, Verifying and Replaying
A message lands in the team channel at 08:50. Finance: “Why do 340 accounts show a paid plan that never paid?”
You start digging. Every one of those upgrades came through the same route, POST /webhooks/billing, the endpoint your payment provider calls when a subscription goes active. The handler is four lines. It reads a customer id off the body, looks up the user, sets the plan to pro, returns 200. It works. It has always worked.
That is the problem. It works for anyone. The URL sits right there in your client bundle. Anyone who can type curl can POST a JSON blob shaped like a subscription event, and your server will cheerfully mark them paid, because nothing in those four lines asks the one question that matters: did this actually come from the provider?
An unverified webhook endpoint is not an integration. It is an unauthenticated UPDATE statement with a public URL. This article is about closing that hole, and about the four or five other bugs that live one layer down once you do.
The shape of a webhook
Flip the usual direction. Normally you call an API: you send a request, you wait, you get a response. A webhook inverts that. Something happens on someone else’s system (a charge clears, a build finishes, a file lands), and they call you. You publish a URL, they POST an event to it, and your job is to accept it and answer fast.
Everything hard about webhooks is hiding in that picture. How do you know step 1 is real? What if the same event arrives twice? What if step 3 is slow? We will take them in order, starting with the only one that is a security bug if you get it wrong.
Verify before you trust
The provider and you share a secret, a random string you both hold and nobody else sees. When the provider sends an event, it computes an HMAC over the request body using that secret and puts the result in a header. You recompute the same HMAC on your side. If your value matches theirs, the body is authentic and untampered, because forging that value requires the secret.
That is the whole idea. HMAC-SHA256 takes (secret, message) and returns a fixed-length digest. Change one byte of the message, or use the wrong secret, and the digest is completely different. It is not encryption, it is a tamper-evident seal.
Providers do not hash the body alone. They prepend a timestamp first, so the signed message is timestamp.rawBody, and the timestamp is signed too. You will see why in a minute.
Here is the check in Node, using the built-in crypto module. No library needed, and understanding the raw version means you will spot the bugs in any SDK too.
import crypto from "node:crypto";
// The header looks like: "t=1737000000,v1=<hex digest>"
function verifySignature(rawBody, sigHeader, secret) {
const parts = Object.fromEntries(
sigHeader.split(",").map((p) => p.split("="))
);
const timestamp = parts.t;
const given = parts.v1;
if (!timestamp || !given) return false;
// Hash the timestamp and the UNTOUCHED body bytes. Never a re-parse.
const hmac = crypto.createHmac("sha256", secret);
hmac.update(timestamp);
hmac.update(".");
hmac.update(rawBody); // rawBody is a Buffer, exactly as received
const expected = hmac.digest(); // a Buffer
const givenBuf = Buffer.from(given, "hex");
if (givenBuf.length !== expected.length) return false;
return crypto.timingSafeEqual(expected, givenBuf);
}Two lines in there are not optional, and both trip people up.
crypto.timingSafeEqual, not ===. A normal string comparison returns the instant it finds a mismatching character. That timing is measurable over enough requests, and it leaks the digest one byte at a time, which is enough to forge a valid signature without ever knowing the secret. timingSafeEqual always looks at every byte, so the response time reveals nothing. It also throws if the two buffers differ in length, which is why the length guard comes first.
The body is a Buffer, and it must be the exact bytes that arrived. That word “exact” is doing a lot of work, and it is where the next section lives.
The raw-body trap
Here is the bug that has cost more engineer-hours than any other on this list. Your framework parses JSON for you. By the time your handler runs, req.body is a tidy object, and the raw bytes are gone. So you do the natural thing: you JSON.stringify(req.body) to get a string back and hash that.
It fails. Every time, and only in production, where the real signatures are.
The signature is computed over the exact bytes the sender serialized. A round trip through a parser does not reproduce those bytes. The sender might have put a space after a colon; JSON.stringify will not. The sender might order keys one way, escape a Unicode character a certain way, or format a number as it pleases. Parse then re-serialize and you get equivalent JSON with different bytes, and a different byte means a different HMAC.
The fix is to keep the original bytes. In Express, mount express.raw on the webhook route before the global JSON parser, so req.body stays a Buffer:
import express from "express";
const app = express();
// Raw body for the webhook route ONLY, mounted before express.json().
app.post(
"/webhooks/billing",
express.raw({ type: "application/json" }),
(req, res) => {
const sig = req.get("webhook-signature");
if (!verifySignature(req.body, sig, process.env.WEBHOOK_SECRET)) {
return res.status(400).send("bad signature");
}
// req.body is a Buffer here. Parse it yourself, AFTER verifying.
const event = JSON.parse(req.body.toString("utf8"));
// ... handle event ...
}
);
app.use(express.json()); // every other route gets normal parsingOrder matters. If app.use(express.json()) runs first, it consumes the body on every route and the raw bytes are gone before your handler sees them. Web-standard runtimes (Workers, Deno, Bun, Hono) sidestep the middleware ordering because you read the body yourself, but they add their own trap:
export default {
async fetch(req, env) {
const raw = await req.text(); // read the body ONCE, as text
const sig = req.headers.get("webhook-signature");
if (!(await verifyWebCrypto(raw, sig, env.WEBHOOK_SECRET))) {
return new Response("bad signature", { status: 400 });
}
const event = JSON.parse(raw); // parse the same string you verified
await env.QUEUE.send({ eventId: event.id });
return new Response("ok");
},
};A Request body is a stream you can read once. Call req.text() and then req.json() and the second one throws “body already used”. Read the text once, verify it, then JSON.parse that same string. Do not reach for req.json() at all on a webhook route.
See it flip
Sign a body with a secret, then change a single character and watch the digest walk away from the header. This runs entirely in your browser with the Web Crypto API, the same HMAC-SHA256 the server uses. Click Sign and fill, hit Verify to see a match, then edit one character in the body and Verify again.
Timestamps stop replay
Verification proves the body is authentic. It does not prove it is fresh. If an attacker captures one genuine, correctly-signed request (a proxy log, a leaked mirror, a misconfigured gateway), they can send those exact bytes to your endpoint again. The signature still validates, because the bytes are unchanged. That is a replay attack, and against a charge.succeeded webhook it means replaying a real payment event to claim the goods over and over.
This is why the timestamp is part of the signed content. You check that it is recent, and because it is signed, an attacker cannot bump it forward without breaking the signature.
const timestamp = Number(parts.t);
const ageSeconds = Math.abs(Date.now() / 1000 - timestamp);
if (ageSeconds > 300) {
return res.status(400).send("timestamp outside tolerance");
}Five minutes is the common default and a reasonable one. It is wide enough to survive clock skew and a slow retry, tight enough that a captured request is useless within minutes. Tighten it for high-value endpoints. Do the timestamp check and the signature check; neither replaces the other.
At-least-once means duplicates
Now the delivery guarantees, which are weaker than most people assume.
Webhook delivery is at-least-once, never exactly-once. If the provider sends an event and does not get a clean 2xx back in time (your server was slow, the connection dropped, a load balancer hiccuped), it assumes failure and sends the same event again. Sometimes the first one actually succeeded and only the acknowledgement got lost. Either way, the same event lands twice, and your handler charges the order or sends the email a second time.
The defense is idempotency keyed on the event id. Every event carries a unique id. Before you act, record that id; if you have seen it, do nothing. A unique constraint plus ON CONFLICT DO NOTHING turns a duplicate into a cheap no-op at the database level, with no race between checking and inserting.
CREATE TABLE processed_events (
event_id text PRIMARY KEY,
created_at timestamptz NOT NULL DEFAULT now()
);async function claimEvent(db, eventId) {
const { rowCount } = await db.query(
"INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING",
[eventId]
);
return rowCount === 1; // true = first time we have seen it, false = duplicate
}Notice the id you dedupe on is the provider’s event id, carried in the body or a header, not one you generate. This is the same money-safe pattern that shows up on the sending side too, covered in depth in Idempotency Keys. One more thing the picture does not show: ordering is not guaranteed either. A subscription.updated can arrive before the subscription.created it depends on. Treat each event as a fact you reconcile against your current state, not a step in a sequence you must replay in order. When order truly matters, refetch the object from the provider’s API and trust that, not the arrival order.
Answer fast, work later
Back to step 3 from the opening diagram. The provider is holding a connection open waiting for your 2xx, and it will not wait long. Stripe gives you around 20 seconds; GitHub is stricter, around 10. Blow past that and the provider records the delivery as failed and schedules a retry, even though your handler is still running and will eventually finish. Now the slow work happens twice. Do that under load and you get a retry storm: every delivery times out, every timeout triggers a retry, and the retries pile onto the same overloaded handler.
The rule: do the smallest durable thing, then acknowledge. Verify, record the event id, hand the payload to a queue, return 200. That is a few milliseconds of work. The actual fulfilment (charging, emailing, calling other services) happens in a worker that is no longer holding the provider’s connection hostage. The dedupe insert doubles as the durable record, so the two concerns collapse into one clean handler:
app.post(
"/webhooks/billing",
express.raw({ type: "application/json" }),
async (req, res) => {
// 1. authenticate
if (!verifySignature(req.body, req.get("webhook-signature"), SECRET)) {
return res.status(400).send("bad signature");
}
const event = JSON.parse(req.body.toString("utf8"));
// 2. dedupe + durably record, in one atomic step
const isNew = await claimEvent(db, event.id);
if (!isNew) return res.sendStatus(200); // seen it, ack and stop
// 3. enqueue the slow part, then ack immediately
await queue.add("process-webhook", { eventId: event.id });
res.sendStatus(200);
}
);Everything expensive now lives behind that queue.add. If your worker crashes mid-job, the job is still on the queue and gets retried by your infrastructure, on your terms, without the provider ever knowing. The machinery for that, durable queues and retriable jobs, is its own topic in Background Jobs and Queues.
What the provider does when you fail
Return the wrong status and you set off machinery you do not control, so it pays to know the contract.
2xxmeans “I have got it, stop sending.” Return this the moment the event is safely recorded, even for events you chose to ignore. An event type you do not handle is still a successful delivery. Answer200and move on.- Non-
2xx(or a timeout) means “I failed, please retry.” The provider backs off on a schedule, typically exponential: seconds, then minutes, then hours, often across two or three days. Stripe retries for up to about 3 days; GitHub does not retry automatically at all but lets you redeliver by hand. - Persistent failure eventually gets your endpoint disabled. After enough consecutive failures the provider stops delivering and emails the account owner. Now you are silently missing every event until someone notices and re-enables it.
This maps onto the same discipline from Status Codes and Errors. Do not return 500 because your downstream was briefly down after you already recorded the event; you will get a duplicate delivery on top of the work already queued. Do not return 200 on a signature failure either, or you teach the provider that garbage was accepted. 400 for “I reject this” (bad signature, malformed), 2xx for “recorded”, 5xx only for “genuinely could not record it, please do resend.”
Testing without a public server
Providers cannot POST to localhost. You have two ways around that.
A tunnel exposes your local port at a temporary public URL. cloudflared tunnel --url http://localhost:3000 prints an HTTPS address you paste into the provider’s dashboard; ngrok http 3000 does the same. Requests hit the public URL and forward straight to your machine, so you can set a breakpoint in a real handler on a real delivery.
Most providers also ship a CLI listener that skips the tunnel: it opens an outbound connection to their servers and forwards events to your local port, often with a command to fire a fake event on demand. That gives you a fresh, correctly-signed test event whenever you want one, which is the fastest inner loop.
And when a real delivery fails, do not reproduce it from scratch. Every decent provider dashboard keeps a log of recent deliveries with a Resend or Redeliver button. Fix the bug, redeliver the exact event, watch it go green.
When you are the sender
Flip it around. The moment your product lets customers receive events from you, everything above becomes your responsibility, pointed outward. You are now the retrier.
async function deliver(endpoint, event) {
const body = JSON.stringify(event);
const ts = Math.floor(Date.now() / 1000);
const signature = signHmac(ts + "." + body, endpoint.secret); // your HMAC
const backoffSeconds = [0, 5, 30, 300, 3600]; // immediate, then widening
for (const wait of backoffSeconds) {
if (wait) await sleep(wait * 1000);
const res = await fetch(endpoint.url, {
method: "POST",
headers: {
"content-type": "application/json",
"webhook-id": event.id,
"webhook-timestamp": String(ts),
"webhook-signature": "v1," + signature,
},
body,
});
if (res.status >= 200 && res.status < 300) return; // delivered, done
}
await flagEndpointFailing(endpoint); // exhausted retries: back off, alert owner
}The obligations mirror the receiving side exactly. Give every event a stable id so your customers can dedupe. Sign the body with a per-endpoint secret and include the signed timestamp. Retry with exponential backoff on any non-2xx, and cap it. Disable endpoints that fail for too long, and tell the owner. Let them replay from a dashboard when they have fixed their side. Because fetch to a customer-supplied URL is an outbound request to the open internet, treat those URLs as untrusted: block private address ranges so a webhook cannot be aimed at your own internal metadata endpoint. That last one is a real class of attack, not a hypothetical.
Adopting the Standard Webhooks header conventions here is worth it. Your customers may already have code that verifies them, and you inherit a spec that got the timestamp, the id, and the key-rotation story right so you do not have to invent them.
Summary
- A webhook is the provider calling you when something happens. An unverified endpoint is an unauthenticated write to your database, so verification comes first, not as a later hardening pass.
- Verify with HMAC over the raw body plus a signed timestamp, using the shared secret, and compare with a constant-time function like
crypto.timingSafeEqual, never===. - The signature covers the exact received bytes. Parse-then-re-serialize changes the bytes and breaks the check, so capture the raw body before any JSON parser runs, and parse only after verifying.
- A signed timestamp with a tolerance (5 minutes is typical) stops captured requests from being replayed.
- Delivery is at-least-once: expect duplicates and dedupe on the provider’s event id with a unique constraint. Do not assume ordering.
- Acknowledge fast. Verify, record, enqueue, return
2xxin milliseconds; do the slow work in a background job so a slow handler cannot trigger a retry storm. - Return
2xxfor “recorded” (even for ignored events),4xxfor “rejected”,5xxonly for “could not record, please resend.” Persistent failure gets your endpoint disabled. - Sending webhooks flips every rule outward: stable ids, signed bodies, backoff, replay, endpoint disabling, and blocking private URLs on the requests you make.