My LibraryBuy PDF
Part 8/Resilience Patterns

Composing Stability Patterns

The postmortem had one line in it that stopped the room: “All five resilience patterns were present in the failing code path.” Timeout, retry, circuit breaker, bulkhead, fallback. Every one of them, added over the previous year, each in its own sprint by a different person, each reviewed and merged and forgotten. And checkout still fell over for nineteen minutes the afternoon the payments provider got slow.

No single pattern was wrong. The retry backed off with jitter. The breaker had a sane threshold. The timeout was tuned to the dependency’s real p99. They just were not arranged. They sat in the code path in the order they happened to be written, wrapping each other by accident, and in the one moment they were supposed to work as a team they worked against each other.

Here is the shape of the accident. The retry was added first, wrapped tightly around the raw call. The breaker was added months later, on the outside of the retry. So the breaker never saw an individual failure. It only saw the final verdict of a whole three-attempt retry cycle, which meant that while the provider was dying, every “one” failure the breaker counted was actually three real calls hammering a service that was already on the floor. The breaker tripped eventually, long after the retries had poured gasoline on the fire. And the fallback? It lived in a try/catch that only wrapped the breaker, not the retry above it, so the retry’s final throw sailed straight past it and became a 500.

Five good patterns. One bad composition. This lesson is about the composition, because that is the part nobody writes down.

The patterns are layers, and the order is the program

The four preceding lessons each solved one failure in isolation. A timeout converts a hang into a fast error. A retry with backoff and jitter rides out a transient blip. A circuit breaker stops you hammering a dependency that is clearly down. A bulkhead caps how much of your resources one dependency can ever eat. A fallback turns the final failure into a degraded-but-working answer.

Read them as a menu and you will reach for one and feel covered. You are not. Each of them leaves a gap that exactly one of the others closes. A timeout with no fallback is a faster 500. A retry with no breaker amplifies an outage. A breaker with no timeout never trips, because a hung call never fails. They are not five choices. They are five layers around a single outbound call, and they only work as a stack.

fallback · catch what is left, degraderetry · backoff + jitter, idempotent onlycircuit breaker · stop repeating an outagebulkhead · cap concurrencytimeout · bound each attemptthe calleach layer catches adifferent failure:timeout → the hangbulkhead → the floodbreaker → the repeatretry → the blipfallback → the resta request flowsinward to the call,an answer flows out
The five patterns nest around one call. Innermost bounds the attempt, outermost catches whatever escapes. Each layer converts or catches a failure the inner ones let through.

That nesting reads, from the inside out, as one expression: fallback(retry(breaker(bulkhead(timeout(call))))). The timeouts-and-bulkheads lesson ended on the inner four of these. The fallback on the outside is what turns “the stack gave up” into “the user still got a page”. Every ordering decision in this article is an argument about which layer belongs where in that expression, and none of them are cosmetic.

Read the stack from the inside out

Start with the accident from the opening, written plainly. Two patterns, wrapped in the order they were added, plus a fallback in the wrong place:

// the raw call to a flaky provider
const charge = (order, signal) =>
  fetch(PAY_URL, { method: "POST", body: JSON.stringify(order), signal });

// added first: retry hugs the call
const withRetries = (order) => retry(() => charge(order), { attempts: 3 });

// added later: breaker on the OUTSIDE of the retry
const guardedCharge = breaker(withRetries);

async function pay(order) {
  try {
    return await guardedCharge(order);
  } catch (err) {
    // this catch only ever sees the breaker's error, and the message is generic,
    // so half the real failures fall through and 500
    if (err instanceof CircuitOpenError) return cachedQuote(order);
    throw err;
  }
}

Three things are broken and all of them are ordering, not logic. The breaker wraps the whole retry loop, so it counts cycles instead of attempts. The retry has no idea the breaker exists, so an open circuit does not stop it; it keeps looping on a dead service. And the fallback only catches CircuitOpenError, so a plain timeout after the last retry becomes a 500.

The fix is to nest them deliberately. A tiny helper makes the order explicit and readable, and it composes right-to-left so the leftmost name is the outermost layer, exactly the way the real libraries read:

// each guard is (fn) => wrappedFn. reduceRight makes the first argument outermost.
const pipe = (...guards) => (fn) => guards.reduceRight((inner, g) => g(inner), fn);

Now the stack is one declaration, and you can read the layers top to bottom knowing the top one runs first and last:

const chargeCard = pipe(
  withFallback(queueForLater),                     // outermost: catch the final failure
  withRetry({ attempts: 3, retryOn: isTransient }),// retry the blip, idempotent calls only
  withBreaker({ failureThreshold: 8, cooldownMs: 15_000 }),
  withBulkhead(20),                                // cap concurrent charges
  withTimeout(800),                                // innermost: bound each attempt
)(charge);

// the call site never learns any of this happened
const receipt = await chargeCard(order, idemKey);

withTimeout, withBulkhead, withBreaker and withRetry are thin adapters over the exact withTimeout, bulkhead and circuitBreaker closures you built in the previous two lessons, plus the backoff from retries and backoff. Nothing new is invented here. The value is entirely in the arrangement.

The one wrapper worth showing in full is the retry, because two lines in it are what make the whole stack cooperate rather than fight:

function withRetry({ attempts, retryOn, base = 200, cap = 2000 }) {
  return (fn) => async (...args) => {
    let lastErr;
    for (let attempt = 1; attempt <= attempts; attempt++) {
      try {
        return await fn(...args);                 // fn is the breaker below us
      } catch (err) {
        lastErr = err;
        if (err instanceof CircuitOpenError) throw err;   // (1) breaker is open: stop now
        if (!retryOn(err) || attempt === attempts) throw err;
        const wait = Math.min(cap, base * 2 ** (attempt - 1)) * Math.random(); // full jitter
        if (wait > budgetLeft()) throw err;               // (2) no time left: do not sleep to fail
        await sleep(wait);
      }
    }
    throw lastErr;
  };
}

Line (1) is the handshake between retry and breaker. An open circuit is not a blip you retry through; it is a definitive “this dependency is down, stop.” So the retry treats CircuitOpenError as fatal and gives up immediately instead of burning its remaining attempts on a breaker that is going to reject every one of them. Line (2) is the handshake with the deadline, and it gets its own section below.

One request, three failures

The point of the stack is that the same wrapped call behaves completely differently depending on what is wrong underneath it, and no caller has to know which. Walk one request through on three different days.

Healthy · the first try landsrequestall layerspassthe callanswers200 OKno layer has to act; on a good day the whole stack is invisibleTransient blip · retry absorbs itrequestattempt 1timeout ✕retry waitsbackoffattempt 2call ✓200 OKthe timeout turns the hang into a fast error; the second attempt landsHard down · breaker to fallbackrequestattempts✕ ✕ ✕breakerOPENfallbackcached200degradedonce the breaker is open the calls stop fast; the fallback serves a cached answer
One wrapped call, three dependency states. On a good day no layer acts. A blip is absorbed by retry. A hard outage trips the breaker and the fallback serves a degraded answer.

Trace the hand-offs, because the hand-off is the whole point of a stack.

On the healthy day, the request passes through all five layers untouched and the call answers on the first attempt. The stack costs you almost nothing. That matters: a resilience stack you can only afford in an emergency is one you will be tempted to skip, and the skipped call is always the one that takes you down.

On the blip day, the first attempt times out. The timeout is what makes this recoverable at all, because it turns a silent hang into a TimeoutError the retry can see. The retry waits a jittered backoff and tries again, the blip has passed, and the call succeeds. The breaker counted one failure and one success and stayed closed. The user saw a slightly slower response and nothing else.

On the hard-down day, the attempts keep failing, the breaker’s counter crosses its threshold and it trips OPEN. Now the interesting thing happens: subsequent calls do not even reach the timeout, because the open breaker rejects them in microseconds. The retry sees CircuitOpenError, recognizes it as fatal, and gives up without looping. The fallback catches that and serves a cached quote. The provider is on fire and the user still gets a working, if slightly stale, checkout.

Why the breaker lives inside the retry loop

This is the ordering decision the opening got wrong, and it is the one people argue about, so it is worth being precise. The breaker sits inside the retry, meaning the retry is the outer layer and each of its attempts passes through the breaker individually. Written out, retry(breaker(call)), not breaker(retry(call)).

The reason is what the breaker gets to count.

retry wraps breaker · rightattempt 1breaker counts fail 1 / 5attempt 2breaker counts fail 2 / 5attempt 3fail 3 / 5attempt 4OPEN → CircuitOpen, instanteach attempt is counted; the loop quits the moment it opensthe sick dependency stops being hit almost immediatelybreaker wraps retry · wrongbreaker (outer) sees one outcomeretry: 3 real calls to the dead dep✕✕✕= 1 failto trip (5 fails) it needs 5 cycles= 15 real calls before it opensthe breaker only sees whole cycles, so it tripsslowly, and hammers the dependency 3x per cycle
Left: retry outside the breaker, so every attempt is counted and an open circuit ends the loop fast. Right: breaker outside the retry, so it only sees whole cycles, trips slowly, and each cycle hammers the dying dependency three more times.

With the breaker inside, each retry attempt is one call through the breaker, so each failure is counted. The breaker’s picture of the dependency’s health stays honest, it trips after the real number of failures, and the instant it opens the next attempt gets a CircuitOpenError that ends the loop. Fast to trip, fast to stop.

With the breaker outside, it only ever sees the verdict of a whole retry cycle. Three real failed calls collapse into a single counted failure. To reach a threshold of five it now needs five full cycles, which is fifteen real calls slamming a service that is already down, and only then does it trip. You have built a machine that waits until the outage is at its worst before it does anything, and triples your traffic against the victim in the meantime. That is precisely the amplification the breaker was supposed to prevent.

One deadline, shared by every layer

A stack that fixes hangs can still blow your latency budget, because the layers add time. Three attempts at an 800ms timeout is a 2.4-second worst case before you even count backoff. If the user, or the caller upstream of you, is only willing to wait 2 seconds, your carefully composed stack is now producing answers for a tab that closed half a second ago, and every one of those doomed attempts is holding resources the next live request needs.

The retry budget has to fit inside the request’s deadline, not the other way around.

Fresh 800ms timeout every attemptattempt 1 · 800msattempt 2 · 800msattempt 3 · pastclient gone at 2000mswork no one readsOne budget, each attempt clamped to what is leftattempt 1 · 800msattempt 2 · clampedtoo little left → fallbackdeadline 2000msthe per-attempt timeout must be small enough that attempts plus backoff fit the budgetwhen the budget cannot fit another try, stop and degrade instead of overrunning
Give each attempt a fresh full timeout and the retries sum past the client's deadline, doing work nobody reads. Carry one budget and clamp each attempt to what is left, and the whole stack finishes inside the deadline, degrading when time runs out.

You carry a deadline, not a duration. Stamp it once at the edge, keep it in an AsyncLocalStorage so it rides along without threading through every signature, and let each layer read what is left. That is the budgetLeft() the retry checked in line (2) above: before sleeping for a backoff it asks whether there is even time to make the sleep worthwhile, and if not, it stops and lets the fallback take over now rather than after a pointless wait.

import { AsyncLocalStorage } from "node:async_hooks";
const deadline = new AsyncLocalStorage();

// at the edge, once
app.use((req, res, next) => deadline.run({ at: Date.now() + 2000 }, next));

// everywhere, no plumbing
const budgetLeft = () => {
  const d = deadline.getStore();
  return d ? Math.max(0, d.at - Date.now()) : Infinity;
};

// the innermost timeout clamps itself to the smaller of its ideal and the remainder
const withTimeout = (ms) => (fn) => (...args) =>
  fn(...args, AbortSignal.timeout(Math.min(ms, budgetLeft())));

Two timeouts, then, not one. A per-attempt timeout on the inside bounds each individual call. A total deadline on the outside bounds the whole operation including every retry and backoff. The pros build both: this is exactly why the timeouts lesson insisted a deadline has to travel, and it is the same reason AbortSignal.any merges the caller’s cancellation with your per-attempt clock. Miss it and your retries quietly turn a fast dependency into a slow one on the exact day it can least afford the extra load.

Watch the whole stack work

Set the dependency to healthy, flaky, or down, then send requests and watch which layer handles each one. Healthy calls sail through on the first try. Flaky calls fail and the retry catches most of them on a second attempt. Turn it fully down and hammer it: the timeout fires on every attempt, the breaker’s count climbs, it trips, and from then on requests fail fast and land on the fallback instead of touching the dead service. Flip it back to healthy and wait for the cooldown to watch the breaker probe and close.

interactiveOne call, the whole stack, three dependency states

The counters tell the story the trace fills in. Under flaky the retried column climbs while fallback stays near zero, because the second attempt usually lands. Under down, watch the first few requests reach the timeout, then the breaker trip, and from that point every request goes straight to fallback without a single call touching the dead dependency. That gap, between calls attempted and calls made, is the load the breaker took off the victim so it could recover.

An untested failure path is just a hope

Here is the uncomfortable truth about everything above. The happy path runs a million times a day and any break in it pages you within minutes. The failure path runs only during an incident, which is to say almost never, which is to say it rots in the dark. Someone changes the shape of an error, the retryOn predicate stops matching it, and now your retry silently gives up on failures it used to absorb. Nobody notices until the provider goes down for real and the stack you were so proud of does nothing, because the one branch that mattered was broken for six months and no test ever ran it.

Resilience is not a library you install. It is a set of decisions about what happens when each thing fails, and a decision you never exercised is not a decision. It is a comment claiming there is a decision. The only way to trust the stack is to break the dependency on purpose and watch the right layer catch it.

checklist for any network callinject the fault, prove the layera deadline on every calladd 5s latency → it times out fastretry only idempotent + transientreturn 500 then 400 → retried, then notbreaker inside the retryhold it down → it trips, calls stopa bulkhead per dependencyflood it → its neighbours keep servinga fallback wired to the failurekill it → degraded 200, not a 500one budget for the whole requesttime the trace → under the deadlineevery transition is observableread the metrics → you saw it tripthe failure path you never inject is the one that fails, silently, in the incident
A defence-in-depth checklist for a network call, and the fault you inject to prove each layer actually fires. The line you never test is the one that fails in the incident.

The cheap version of this fits in your test suite. Force each dependency to fail and assert the right layer answers: mock a 5-second hang and assert the call returns a TimeoutError in under a second, mock a run of 503s and assert the breaker opens and the fallback serves, mock a 400 and assert the retry does not fire. These tests are the only proof that the wiring you drew on the whiteboard is the wiring that shipped.

The expensive version is worth more and you graduate to it once the cheap one is green: fault injection and game-day exercises against a real environment. Deliberately add latency to a live dependency, kill an instance, trip a breaker by hand, and watch your dashboards during business hours with the whole team present, so you discover on a calm Tuesday that the fallback rotted rather than at 3am during an actual outage. None of this works if you cannot see the layers act, which is why every transition emits a metric or a span. Wire the stack into your observability and, for model calls, your tracing, from the first day. A breaker you cannot watch trip is a breaker you cannot trust.

Do not wrap everything in five layers

The failure mode of a lesson like this is a junior engineer who now puts all five layers around every function call in the codebase, including the pure ones. Resist it. The full stack is expensive to build, expensive to reason about, and expensive to test, and most calls do not need it.

The full five layers earn their place only when all of these are true: the call crosses a boundary you do not control, that boundary can be down independently of your process, the call can hang, and you have a real fallback to offer when it fails. A charge to a payment provider qualifies. A read from your own in-memory cache does not; wrapping a synchronous function in a circuit breaker is nonsense, because there is nothing to fail fast toward. Fix the bug instead.

Between those extremes, take what you need and skip the rest. A read from a well-run internal service on the same network might deserve a timeout and a fallback and nothing more. A best-effort analytics beacon deserves a timeout and a shrug. And the two anti-patterns from the earlier lessons still bite hardest here: one global breaker shared across five dependencies couples them so that any one dying takes the other four offline, and one global bulkhead around every outbound call recreates the exact resource coupling you were trying to break. Scope one breaker and one bulkhead per dependency, always. A shared compartment is not a compartment.

Summary

  • The resilience patterns are not a menu you pick from. They are layers around one call, and each closes a gap the others leave open. A timeout with no fallback is a faster 500; a retry with no breaker amplifies an outage; a breaker with no timeout never trips.
  • The whole stack is one expression: fallback(retry(breaker(bulkhead(timeout(call))))). Innermost bounds the attempt, outermost catches what escapes. The order is not cosmetic. It is the failure behavior.
  • The breaker lives inside the retry loop. With retry outside, each attempt is counted, the breaker trips honestly, and an open circuit ends the loop fast. With the breaker outside, it only sees whole cycles, trips slowly, and hammers the dying dependency three times per cycle on the way. This is the ordering the opening got wrong.
  • Retry only what is safe and worth repeating. Idempotent operations (carry an idempotency key for writes), transient failures only (5xx and timeouts, never a 4xx), and stop the moment you see a CircuitOpenError.
  • One deadline, carried, not many durations invented. A per-attempt timeout on the inside and a total budget on the outside. Each attempt clamps to what is left, and when the budget cannot fit another try, you degrade now instead of overrunning the deadline doing work nobody reads.
  • The same wrapped call behaves differently under each failure, and no caller has to know which. Healthy: nothing acts. Blip: retry absorbs it. Hard down: the breaker trips and the fallback serves. That hand-off between layers is the entire design.
  • An untested failure path is a hope, not a safety net. Inject the fault that each layer is supposed to catch, cheaply in tests and expensively in game-days, and wire every transition into your observability so you can watch it work. The branch you never exercise is the one that fails in the incident.
  • Do not wrap everything in five layers. The full stack is for a call across a boundary that can be down independently, can hang, and has a real fallback. Scope one breaker and one bulkhead per dependency; a shared one recouples what it was meant to isolate.
  • Design for failure, then test it. Resilience is not a library you bolt on. It is a set of decisions about what happens when each thing breaks, and the only ones you can trust are the ones you have deliberately broken on purpose.