My LibraryBuy PDF
Part 5/Scale and Reliability

Retries, Backoff and Dead Letters

A dependency you call had a bad two seconds. Maybe a deploy, maybe a leader election, maybe one long garbage-collection pause. Two seconds. Your service noticed the errors and did the sensible thing: it retried. So did every other service calling that dependency. So did their retries.

When the dependency came back up, it was not met with its normal traffic. It was met with normal traffic plus a wall of retries that had been piling up during those two seconds, all released at once. It fell over again. The two-second blip is now a forty-minute outage, and it clears only when someone drains a queue by hand at 3am.

That is a retry storm, and the uncomfortable part is that every client in it was doing what the textbook says. Retrying is easy. Retrying without turning a small blip into an outage takes more care than the naive version, and the gap between the two is where this article lives.

What is even retryable

Before you retry anything, you answer one question: would trying again plausibly work? If the answer is no, a retry is not resilience. It is the same failure, delayed and multiplied.

Failures split cleanly into two buckets.

Transient failures are timing accidents. A connection got reset, a DNS lookup blipped, a server was briefly overloaded and shed your request, a database transaction lost a race. Nothing about your request was wrong. Try again in a moment and it may well succeed.

Permanent failures are deterministic. You sent a malformed body, you are not authenticated, the resource does not exist, the input failed validation. Retrying a 400 does not fix the 400. You will just fail the same way four more times, faster, and burn a chunk of your error budget doing it.

Here is the practical table for HTTP and Postgres, the two you will hit most:

Response or error Retry it?
400, 401, 403, 404, 422 No. Deterministic; it fails identically next time
408 Request Timeout, 429 Too Many Requests Yes, with backoff (and honour Retry-After)
500 Internal Server Error Maybe. Could be a blip, could be a bug you will hammer
502, 503, 504 Yes. A hop or the origin is briefly unhealthy
ECONNRESET, ETIMEDOUT, ECONNREFUSED, EAI_AGAIN Yes. Transient network faults
Postgres 40001, 40P01 Yes, but retry the whole transaction

Those two Postgres codes are worth calling out. 40001 is serialization_failure and 40P01 is deadlock_detected. Both mean two transactions raced and the database picked a loser to abort. That loser is supposed to try again; it is the documented contract of running at REPEATABLE READ or SERIALIZABLE. The catch is that once any error in the 40 class fires, the transaction is poisoned and every further command is rejected until you ROLLBACK. So you cannot retry the failed statement. You retry the entire transaction from BEGIN. See Transactions and Isolation Levels for why the race happens in the first place.

The 500 row is the honest one. A 500 might be a transient hiccup, or it might be a null-pointer bug that will fire every single time. You cannot tell from the outside. Retry it a small number of times and move on. Do not build a system that hammers a genuinely broken endpoint hoping it heals.

Only retry what is safe to repeat

Transient-vs-permanent is half the gate. The other half is nastier, because it is about a failure mode you cannot see.

Picture a POST /charges that moves money. You send it, and the connection times out waiting for the response. Now answer this: did the charge go through?

You genuinely do not know. The request may have never reached the server. Or it reached the server, created the charge, and the response got lost on the way back. From where you sit, a lost request and a lost response look identical: a timeout. Retry blindly and in the second case you have just charged the customer twice.

This is why retries and idempotency are inseparable. An operation is idempotent when doing it twice has the same effect as doing it once. Reads are idempotent for free. PUT and DELETE are idempotent by design (setting a value to X twice, deleting a thing twice). POST and PATCH are not, and those are exactly the operations that change money, send email, and create rows.

You have two ways to make a POST safe to retry:

  1. Attach an idempotency key, a unique id the client generates once per logical operation and reuses on every retry. The server records the key, and on a repeat it returns the first result instead of running the work again. This is the whole subject of Idempotency Keys for Money-Safe Endpoints, so I will not re-derive it here.
  2. Design the operation to be naturally idempotent: upsert on a natural key instead of insert, use “set status to shipped” instead of “increment shipment count.”

If you cannot do either, do not retry the write. Surface the ambiguity to something that can reconcile it (a human, a reconciliation job) rather than guessing.

a request failedIs the failure transient?429, 503, timeout, deadlockIs it safe to repeat?idempotent, or has a keyAny budget left?attempts, deadline, quotaretry after backoff + jitteryesyesyesnononofail fast (4xx)surface the errordead-letter
Every failed request passes through the same three gates. Miss any one and the answer is not retry.

The mechanics: fixed, exponential, jitter

Say the failure passed all three gates and you are going to retry. When do you retry? This is the part people get wrong, and it is the part that decides whether your retries help or hurt.

Fixed delay is a trap

The first thing everyone reaches for is a fixed pause. Wait one second, try again. Wait one second, try again. It is simple, and it is quietly dangerous for two reasons.

It gives a struggling service no room. If the dependency is overloaded, a steady drumbeat of retries every second keeps it overloaded. And worse, it synchronizes. Every client that failed at the same instant waits the same one second and retries at the same next instant, together, forever. You have built a metronome that periodically slams the service with your entire failed population at once.

Exponential backoff spreads the attempts out in time

The fix for the first problem is to back off more each time you fail. Double the wait on every attempt:

// attempt 0 -> 1s, attempt 1 -> 2s, attempt 2 -> 4s, attempt 3 -> 8s ...
const BASE = 1000; // ms
const CAP = 30_000; // ms, never wait longer than this

function backoffDelay(attempt) {
  return Math.min(CAP, BASE * 2 ** attempt);
}

Now a persistent problem gets exponentially more breathing room. Two failures in a row and you are waiting seconds, not milliseconds. The CAP matters: without it, attempt 12 would ask you to wait over an hour, which is never what you want.

This is genuinely better. But it does not fix synchronization, and that surprises people. If a thousand clients all fail at t=0, they all compute a one-second delay for their first retry, so they all retry at t=1s. Together. Exponential backoff moved the spike and spaced the spikes apart, but each spike is still the whole herd arriving at the same instant.

Jitter is the piece that actually saves you

The real fix is randomness. Instead of waiting exactly the backoff delay, wait a random amount up to it. This is called full jitter, and it is one line:

function fullJitter(attempt) {
  const ceiling = Math.min(CAP, BASE * 2 ** attempt);
  return Math.random() * ceiling; // uniform between 0 and the backoff ceiling
}

Now those thousand clients each pick their own random delay in 0..1s. Instead of a spike at t=1s, their retries are smeared evenly across the whole second. The recovering service sees a gentle, roughly flat trickle it can actually absorb, instead of a wall. That difference is the difference between a service that recovers and a service that gets knocked straight back over.

1000 clients fail at t=0. Height = how many retry in that slice of time.Fixed 1s / 1s / 1s1s2s3sExponential 1s / 2s / 4s1s3s7sExponential + full jittersame retries, spread across the whole window, no single spike
A thousand clients all fail at t=0. Fixed and plain exponential make them retry in lockstep, forming spikes. Full jitter smears the same retries into a low carpet the service can absorb.

You can feel the difference. Drag the slider up to a busy fleet and toggle jitter off: the retries pile into towers that blow past what the recovering service can handle. Toggle jitter on and the same load flattens into something survivable.

interactiveRetry storm: with and without jitter

Caps, deadlines and budgets: knowing when to stop

Backoff decides how long between attempts. Three other limits decide when to give up entirely, and you want all three.

Cap the per-attempt delay. You already saw this: Math.min(CAP, ...). Attempt ten should not ask for a seventeen-minute nap.

Cap the number of attempts. Three to five is the usual range. The AWS SDKs default to three total attempts, which is one try plus two retries. More than five rarely helps; if four honest tries with backoff did not work, the thing is probably down, and attempts six through ten are just you adding to the pile.

Set an overall deadline, and prefer it to the attempt count. This is the one people skip. “Retry up to 5 times” sounds like a limit, but with exponential backoff five attempts could stretch to a minute or more, and by then the user gave up and the answer does not matter. A deadline says: “keep trying until 15 seconds have passed, then stop, whatever the attempt number.” It composes cleanly with everything upstream, because the caller usually has a deadline too.

async function withRetry(fn, { maxAttempts = 4, deadlineMs = 15_000 } = {}) {
  const giveUpAt = Date.now() + deadlineMs;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const last = attempt === maxAttempts - 1;
      if (last || !isRetryable(err)) throw err;
      const wait = fullJitter(attempt);
      if (Date.now() + wait > giveUpAt) throw err; // deadline wins over the attempt count
      await sleep(wait);
    }
  }
}

Retry budgets, because retries at one layer are not the only retries

Here is the failure the per-caller limits above do not catch. Your service retries three times. But your service is itself being called by something that retries three times. And it calls a database client that retries three times. Nobody coordinated, and the retries multiply.

1 click= 1gateway x3= 3service x3= 9database x3= 271 x 3 x 3 x 3 = 27 hits on a service that is already sick
One user click, three layers, three attempts each. The database sees up to twenty-seven requests. Nobody wrote a bug; the layers just multiplied.

Three layers, three attempts each, and one user action can hit the bottom of the stack twenty-seven times. That is the arithmetic behind most retry storms. The dependency was already struggling, which is why things were failing, and the retries you added to help are what finished it off.

Two rules keep this in check.

Retry at one layer, not every layer. Pick the layer with the best context (usually the one closest to the failing dependency) and turn retries off everywhere else, or make the inner layers fail fast and let the chosen layer own the retrying. Stacked retries are almost never what you meant.

Give each service a retry budget. The idea from Google’s SRE practice is to cap retries as a fraction of live traffic rather than per request. A common figure is 10%: a service will retry only while retries are under 10% of its outgoing requests, and above that it fails fast instead. That single rule turns unbounded amplification into a bounded 1.1x in the normal case, and it is what stops a partial outage from becoming a total one. The AWS SDKs implement a version of this as a retry token bucket: every retry spends a token, successes refill the bucket, and when it runs dry the SDK stops retrying even if it has attempts left.

Respect Retry-After

Sometimes the server does the math for you. A 429 Too Many Requests or a 503 Service Unavailable can carry a Retry-After header telling you exactly how long to wait. When it does, that number wins over your backoff. The server knows something you do not, like when its rate-limit window resets or when maintenance ends.

Retry-After comes in two shapes: a number of seconds, or an HTTP date. Parse both, and cap the result so a hostile or buggy upstream cannot park your worker for a day:

function retryAfterMs(response) {
  const header = response.headers.get("retry-after");
  if (!header) return null;

  const seconds = Number(header);
  const ms = Number.isNaN(seconds)
    ? Date.parse(header) - Date.now() // HTTP-date form
    : seconds * 1000;                 // delta-seconds form

  if (!(ms > 0)) return null;
  return Math.min(ms, 60_000); // never honour more than a minute from a header
}

Then prefer it when present, and fall back to jittered backoff when it is absent:

const wait = retryAfterMs(response) ?? fullJitter(attempt);

The header side of this (emitting 429 and Retry-After so your callers back off) is covered in Rate Limiting and Abuse Control and Status Codes and an Error Contract Clients Can Trust. Here you are on the receiving end, and the etiquette is simple: if a server tells you when to come back, come back then, not sooner.

Timeouts: the precondition for all of it

None of this works without timeouts, and this is the quietest failure of the lot.

A retry needs a failure to react to. If a request hangs, there is no failure. There is no error, no rejected promise, no status code. There is a socket sitting open, waiting for a response that a dead server is never going to send, and your beautiful retry logic never runs because nothing ever told it the attempt was over. You do not get retries. You get a hang, and then another request hangs behind it, and you are out of connections.

So every network call gets a timeout. In modern Node and browsers, AbortSignal.timeout() makes it a one-liner:

// give each attempt 3 seconds, then abort and let the retry logic take over
const res = await fetch(url, { signal: AbortSignal.timeout(3000) });

When the timeout fires, fetch rejects with an AbortError, which your isRetryable check should treat as retryable (a timeout is a transient failure by definition). Two things to keep straight:

  • A per-attempt timeout bounds one try. It is what turns a hang into a retryable error.
  • The overall deadline from earlier bounds the whole operation. Per-attempt timeout plus deadline plus attempt cap is the full set of brakes.

Set the per-attempt timeout shorter than the deadline, obviously, or you can never fit a second attempt in. And set it based on real latency, not a guess: if p99 for this call is 800ms, a 3s timeout is generous; a 30s timeout means you will wait 30 seconds to discover something that was never going to answer.

Here is the whole caller-side pattern in one piece: retryable check, per-attempt timeout, Retry-After, full jitter, deadline, attempt cap.

const RETRYABLE_STATUS = new Set([408, 429, 500, 502, 503, 504]);
const RETRYABLE_CODES = new Set(["ECONNRESET", "ETIMEDOUT", "ECONNREFUSED", "EAI_AGAIN"]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

function isRetryable(err) {
  if (err.name === "AbortError") return true;      // our per-attempt timeout fired
  return RETRYABLE_CODES.has(err.code);            // transient network fault
}

async function fetchJSON(url, { maxAttempts = 4, deadlineMs = 15_000, perTryMs = 3000 } = {}) {
  const giveUpAt = Date.now() + deadlineMs;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    let wait;
    try {
      const res = await fetch(url, { signal: AbortSignal.timeout(perTryMs) });
      if (res.ok) return await res.json();
      if (!RETRYABLE_STATUS.has(res.status)) throw new Error(`HTTP ${res.status}`);
      wait = retryAfterMs(res) ?? fullJitter(attempt);
    } catch (err) {
      if (!isRetryable(err)) throw err;
      wait = fullJitter(attempt);
    }

    if (attempt === maxAttempts - 1) break;          // out of attempts
    if (Date.now() + wait > giveUpAt) break;         // out of time
    await sleep(wait);
  }
  throw new Error(`exhausted retries for ${url}`);
}

And the database version, which retries the transaction, not the statement, because a 40001/40P01 poisons everything after it:

const RETRYABLE_PG = new Set(["40001", "40P01"]); // serialization_failure, deadlock_detected

async function withTxRetry(pool, run, { maxAttempts = 5 } = {}) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const client = await pool.connect();
    try {
      await client.query("BEGIN");
      const result = await run(client);
      await client.query("COMMIT");
      return result;
    } catch (err) {
      await client.query("ROLLBACK").catch(() => {});
      if (!RETRYABLE_PG.has(err.code) || attempt === maxAttempts - 1) throw err;
      await sleep(Math.random() * 50 * 2 ** attempt); // full jitter, small base
    } finally {
      client.release();
    }
  }
}

// usage: the work inside always uses parameterised SQL
await withTxRetry(pool, async (client) => {
  await client.query("UPDATE accounts SET balance = balance - $1 WHERE id = $2", [amount, fromId]);
  await client.query("UPDATE accounts SET balance = balance + $1 WHERE id = $2", [amount, toId]);
});

When you finally give up: dead letters

Retries run out. The deadline passes, the attempts are spent, the budget is dry. Now what?

For a synchronous request, “now what” is easy: you return an error to your caller and let them decide. You cannot hold a live HTTP request open forever, so you fail it honestly and log enough to debug.

Asynchronous work is different, and more dangerous. A job pulled off a queue, a webhook you are delivering, an event you are processing: there is no user waiting on the other end of a socket, so when it exhausts its retries you have a choice, and one of the choices is silent data loss. If you just drop the job, the money never moves, the email never sends, the record never updates, and nobody finds out until a customer complains weeks later.

The answer is a dead-letter queue. When an item fails its last retry, you do not drop it and you do not loop on it forever. You move it to a side queue, out of the hot path, where it waits for a human. A single message that fails every time (a “poison message”, maybe a malformed payload or a bug in your handler) can otherwise wedge a worker or spin retries until the heat death of the universe. The DLQ is where it goes to be looked at instead.

one item, four attempts, growing jittered gapstry 1try 2try 3try 4~1s~2s~4sdead-letteralert on depth > 0on-callinspect + replaythe queue is not the fix. a human looking at the queue is the fix.
Attempts fail with growing jittered gaps until the budget is spent, then the item is parked in a dead-letter queue for an on-call engineer, with an alert firing the moment the queue is non-empty.

Most queue systems give you this out of the box, and the mechanics (maxReceiveCount, failed sets, first-class dead-letter queues) belong to Background Jobs and Queues. What belongs here is the discipline around it.

A dead-letter queue nobody watches is a data-loss queue with extra steps. You did not lose the data outright, which feels better, but if no alert fires and no dashboard shows the depth, the outcome is identical: the work never gets done, and you find out from an angry customer. An empty DLQ is a healthy system. A DLQ that is filling up is your system quietly telling you something is broken, and if nobody is listening, “quietly” becomes “silently,” and silent is the whole problem.

So the rule is short. Graph your DLQ depth. Alert when it goes above zero, or above whatever floor is normal for you. Make replaying or discarding a dead-lettered item a one-command operation, because the fastest way to guarantee nobody ever drains the DLQ is to make draining it painful.

Summary

  • Retry only transient failures. 429, 503, 502, 504, timeouts, connection resets, and Postgres 40001/40P01 are worth another try. 400, 401, 403, 404, 422 are not; retrying them just fails faster and adds load.
  • Retry only what is safe to repeat. A timed-out write is ambiguous: it may have happened. Make it idempotent with an idempotency key or a natural upsert, or do not retry it.
  • Fixed delay synchronizes and hurts. Exponential backoff gives a struggling service room, but plain exponential still makes the whole herd retry in lockstep. Full jitter (random(0, min(cap, base·2^attempt))) is the piece that spreads retries out and prevents the self-inflicted spike.
  • Cap everything. Cap the per-attempt delay, cap the attempt count (3 to 5), and prefer an overall deadline to a raw attempt count.
  • Watch amplification. Three layers retrying three times each is twenty-seven requests at the bottom. Retry at one layer, and give each service a retry budget (a ~10% fraction of traffic) so a partial outage cannot amplify into a total one.
  • Honour Retry-After when a server sends it, and cap what you will honour so a bad header cannot park you for hours.
  • A timeout is the precondition. Without a per-attempt timeout, a hang produces no error, so the retry never fires. AbortSignal.timeout(ms) on every network call.
  • Dead-letter what is left, then actually watch it. Park exhausted async work in a DLQ instead of dropping it, graph the depth, and alert on it. An unwatched DLQ loses data with extra steps.