Circuit Breaker
The tax service didn’t go down. It got slow. That turned out to be worse.
Checkout called an internal pricing API on every order to compute tax. Normally it answered in 40 milliseconds and nobody thought about it. One afternoon a bad deploy on their side dragged its p99 up to about 25 seconds. Not an error. No exception, no 500, just a call that sat there and eventually answered. Within four minutes our checkout service, which had nothing wrong with its own code, stopped serving anything at all. Not just checkout. The cart page, the health check, everything.
Here is how a healthy service kills itself over somebody else’s bug.
Every checkout request called getTax and waited. With no timeout, it waited the whole 25 seconds. Node’s event loop was fine, but the event loop was never the scarce thing. The scarce things were the connection pool to that API, the database transaction each request had opened before it got to the tax step, and the memory each in-flight request pinned while it hung. Requests stacked up. The pool saturated. New requests queued behind the stuck ones and started failing to get a database connection, because the stuck requests were holding those too. Latency on endpoints that never touched tax climbed anyway. The load balancer’s health check timed out, so it pulled the instance, so its traffic shifted onto the siblings, which filled up faster and got pulled next.
The instinct that makes it worse
Look at the code that was running. Someone had already made it “resilient” by adding retries:
async function getTax(cart) {
for (let attempt = 0; attempt < 3; attempt++) {
try {
const res = await fetch(TAX_URL, {
method: "POST",
body: JSON.stringify(cart),
});
return await res.json();
} catch {
await sleep(200 * attempt); // "just try again"
}
}
throw new Error("tax unavailable");
}Read it in isolation and it looks careful. In an outage it is gasoline. When the tax API is healthy, this makes one call. When it is down, this makes three, each one waiting on a service that is already drowning, each one holding your connection three times as long. Multiply that by every checkout in flight and your retry logic is now a load test against the exact service you needed to recover. Retries are for the transient blip, the one dropped packet. They are precisely wrong for a sustained outage, and retries and backoff is where that line gets drawn.
The fix is not a better retry. The fix is to notice the service is down and stop calling it for a while. Which is the whole idea.
Trip like a breaker
The name is borrowed from your fuse box. When a circuit draws too much current, the breaker trips and cuts the line. It does not keep pushing power into a fault and hope. It opens, the house is safe, and later you flip it back on to see if the problem is gone.
A software circuit breaker sits in front of one dependency and watches the calls go through it. While they mostly succeed, it stays out of the way. Once failures cross a threshold, it trips: for a cooldown period it stops calling the dependency at all and fails every request immediately. No connection taken, no timeout waited, no thread parked. Then it lets a single trial call through to check whether the thing has recovered, and either closes back up or trips again.
That behavior is three states and the moves between them. This is the entire pattern.
Three things carry the whole design, and each is a knob you set:
- When to trip. How many failures, or what failure rate, before CLOSED becomes OPEN.
- How long to stay open. The cooldown that gives the dependency room to breathe before you poke it again.
- How to probe. In HALF-OPEN, you let a small number of trial calls through and let their result decide. Succeed and you close. Fail and you are open again, cooldown restarted.
Build one, it is a closure
You do not need a class or a library to understand this. A breaker holds a little private state and wraps one function. That is exactly what a closure is for. Here is a real one, small enough to read in a sitting and correct enough to ship.
class CircuitOpenError extends Error {
constructor() {
super("circuit is open");
this.name = "CircuitOpenError";
}
}
function circuitBreaker(call, {
failureThreshold = 5, // trip after this many failures in a row
cooldownMs = 10_000, // stay open this long before probing
isFailure = () => true, // which errors count as "the dependency is sick"
onStateChange = () => {}, // emit a metric / log line on every transition
} = {}) {
let state = "closed";
let failures = 0;
let openedAt = 0;
let probing = false;
const to = (next) => {
if (next !== state) { state = next; onStateChange(next); }
};
return async function fire(...args) {
if (state === "open") {
if (Date.now() - openedAt < cooldownMs) throw new CircuitOpenError();
to("half-open"); // cooldown is up, time to test the water
}
if (state === "half-open" && probing) throw new CircuitOpenError();
if (state === "half-open") probing = true;
try {
const result = await call(...args);
failures = 0;
probing = false;
to("closed"); // any success is a clean bill of health
return result;
} catch (err) {
probing = false;
if (!isFailure(err)) throw err; // not the dependency's fault, don't count it
failures++;
if (state === "half-open" || failures >= failureThreshold) {
openedAt = Date.now();
to("open");
}
throw err;
}
};
}Every line earns its place. The if (state === "open") block at the top is the fast path: when the breaker is tripped, you throw before touching call, which is the entire point of the pattern. The probing flag is the part beginners skip and then wonder why HALF-OPEN doesn’t help: without it, the moment the cooldown expires every queued request becomes a “trial” and they all slam the maybe-still-dead dependency at once. One probe at a time. Everyone else keeps failing fast until the probe reports back.
Using it looks like wrapping the raw call once and then forgetting the breaker exists:
const rawGetTax = (cart) =>
fetch(TAX_URL, { method: "POST", body: JSON.stringify(cart) })
.then((r) => (r.ok ? r.json() : Promise.reject(new Error("tax " + r.status))));
const getTax = circuitBreaker(rawGetTax, {
failureThreshold: 5,
cooldownMs: 10_000,
onStateChange: (s) => metrics.gauge("tax.breaker", s),
});
// call site never changes
const tax = await getTax(cart);The breaker is a small state machine, which is not a coincidence: the tidy version of “closed / open / half-open with rules between them” is exactly the state-machine pattern, and modelling it that way is what keeps the transitions honest instead of a nest of booleans.
Failing fast is the feature
It feels backwards the first time. You added code whose job is to make more requests fail, on purpose, and that is an improvement. It is, because of what those fast failures buy back.
When the breaker is open, a request to the dead dependency returns in under a millisecond and lets go of everything it was holding: the connection slot, the database transaction, the memory, the worker’s attention. Those resources go back in the pool instead of sitting hostage behind a 25-second timeout. Your service stays responsive for the 90% of traffic that had no business being taken down by the tax API in the first place.
This is why the breaker belongs on the resource side of your thinking, right next to connection pooling and server concurrency. The failure it prevents is not “the tax call errored”. Tax calls were going to error either way. It prevents “the tax call held a connection your login page needed”. Fail fast and the connection comes back.
The cascade it stops
Now zoom out past one service. Real systems are chains. Your API calls a pricing service, which calls a currency service, which calls a rates provider. When the deepest one dies, the failure does not stay put. It climbs.
Without breakers, service C dies, and B fills up waiting on C exactly the way your checkout filled up waiting on tax, and then A fills up waiting on B. One outage at the bottom becomes three outages up the stack, and the users at the top see a dead product. The blast radius is the whole chain.
Put a breaker on B’s call to C and the story ends at the first hop. B’s breaker trips, B stops waiting on C, B returns a fallback fast, and B stays healthy. A never notices. The users get a slightly degraded page instead of a dead one. The breaker’s real job is not protecting B from C. It is protecting everyone above B from B’s reaction to C. That containment is why it is a load-bearing part of a resilient chain, and it composes with the other patterns in background jobs and queues and rate limiting rather than replacing them.
Pair it with a fallback
An open breaker throws fast. Fast is good for your resources, but a raw thrown error is not good for the person waiting on the page. The breaker protects you. A fallback protects them. You want both, and they live on opposite sides of the same try.
async function quoteTax(cart) {
try {
return await getTax(cart); // breaker-wrapped, may throw fast
} catch (err) {
if (err instanceof CircuitOpenError || err.name === "TimeoutError") {
return await cachedTaxFor(cart) // last known-good rate
?? { rate: DEFAULT_RATE, estimated: true }; // safe default, flagged
}
throw err; // a real bug: let it surface
}
}There is a menu of fallbacks, roughly in order of how much you like them:
A cached value is the nicest: the customer sees the last known-good tax rate and never knows anything broke. Caching layers already keep that value around, so an open breaker is often just “read the cache and skip the origin”. A safe default trades correctness for availability, and you flag it so a later job can reconcile. Degraded mode drops the feature entirely (hide the recommendation strip, checkout still works). A clear error is last, but a fast honest error still beats a 25-second hang that ends in the same error. Choosing among these per dependency is the same routing decision covered in fallbacks and routing.
Thresholds, and how to get them wrong
The three knobs are where breakers are won and lost. Set them badly and the breaker either does nothing or makes things worse.
Start with when to trip. There are two common shapes, and they suit different services.
The count breaker (like consecutiveBreaker in the off-the-shelf libraries) is simple and twitchy. Five errors in a row and it opens. Fine for a low-traffic call where five in a row genuinely means “broken”, annoying for a busy endpoint where five in a row happens during any brief network hiccup. The rolling-window breaker (a sampling breaker) opens on a failure rate across the last N calls or the last few seconds, so an isolated blip washes out and only a real problem crosses the line. For anything with meaningful traffic, prefer the window.
Then how long to stay open. Too short and you probe a still-dead service constantly, which both wastes the probe and, if you are unlucky with timing across many instances, recreates the thundering herd you were avoiding. Too long and you keep serving fallbacks for minutes after the dependency came back. Start around 10 to 30 seconds and add a little random jitter so a fleet of instances does not all probe on the same tick.
Watch one trip
Set the dependency healthy or failing and send requests through the breaker. Watch the state, the failure count, and which calls actually reach the dependency versus which fail fast. Flip it to failing and hammer it: the breaker trips after three failures and stops touching the dependency entirely. Flip it back to healthy and wait for the cooldown: one probe goes out, succeeds, and the breaker closes. The logic here is the same circuitBreaker from above, cooldown shortened so you do not have to wait.
Two runs to try. Flip to failing, turn on auto traffic, and watch the counters: reached dep climbs to three, the breaker trips, and from then on failed fast climbs instead while reached dep holds. That gap is the load you took off the dying service. Then flip back to healthy and wait about four seconds. The next send goes half-open, the probe lands, and the breaker closes. Notice you never see a burst of probes hit at once, because only one is allowed through.
Where it belongs, and where it doesn’t
A breaker goes around a call across a boundary you do not control: an external API, another service, sometimes the database or a cache cluster. The test is whether the thing can be down independently of your process and whether a call to it can hang. If both are true, it is a candidate.
That rules a lot out, and being honest about the misfits matters as much as the fits.
Do not wrap a pure in-process function. If it can only fail because of a bug in your own code, there is nothing to fail fast toward. Retrying or breaking a synchronous calculation is nonsense. Fix the bug.
Do not use one global breaker for everything. This is the mistake that turns a helper into a hazard. If a single breaker guards calls to five different services, one of them dying opens the breaker and takes the other four offline with it. Scope one breaker per dependency, keyed by host or even by endpoint if their health differs. A breaker’s whole promise is isolation, and a shared breaker throws that away.
Do not stack a breaker under an eager retry without thinking. Retries and breakers are partners, but a retry loop inside an open breaker either bangs on the open breaker pointlessly or, worse, keeps enough traffic flowing to defeat the trip. The clean arrangement is timeout on the call, breaker around it, and retry (with backoff and jitter) as the outermost layer that gives up quickly once the breaker is open. When you do retry a write, make it safe to repeat with an idempotency key.
A too-sensitive breaker costs you availability you did not need to spend. A hair-trigger count of one or two on a chatty service will flap open on every network blip, and now you are serving fallbacks during moments the dependency was actually fine. A too-sluggish one (huge threshold, long window) trips only after the cascade has already spread, which is the same as not having it. Both failure modes are invisible until you are watching the state transitions, which is the entire reason onStateChange emits a metric. A breaker you cannot see is a breaker you cannot tune, so wire it into your observability from day one.
Summary
- When a dependency is failing, the worst thing your code can do is keep calling it. Retrying a sustained outage amplifies it and holds your own resources hostage. A slow dependency is more dangerous than a dead one, because a held-open call is what exhausts you.
- A circuit breaker is a small state machine in front of one dependency: closed (calls pass, failures counted), open (calls fail fast, dependency left alone to recover), half-open (a single probe decides whether to close or reopen).
- In JavaScript it is a closure wrapping one function, not a class hierarchy. The load-bearing lines are the fast-fail when open and the single-probe guard in half-open.
- A breaker without a timeout is half a breaker. It only trips on failures it can see, and a call that hangs never fails. Give every wrapped call a timeout (
AbortSignal.timeoutis the one-liner) so a slow dependency produces countable failures. - Failing fast is the feature. The fast error returns the connection, transaction, and memory the hung call was holding, keeping your service alive for the traffic that never needed the sick dependency. This is a resource-protection pattern first.
- It stops cascades. Put a breaker at each hop and an outage at the bottom of a call chain is contained at the first hop instead of climbing to the users. The breaker protects everyone above you from your reaction to a failure below you.
- Pair it with a fallback. The breaker protects you; the fallback protects the user. Serve a cached value, a safe default, a degraded mode, or at worst a fast clear error, never a hang.
- Tune the three knobs. Prefer a rolling failure-rate window over a twitchy consecutive count for busy services. Keep the open cooldown in the tens of seconds with jitter. Only count real health failures (
5xx, timeouts), never the dependency’s honest4xx. - Use it when a call crosses a boundary that can be down independently and can hang, and you have a sane fallback. Avoid it when the call is a pure in-process function, when a single global breaker would couple unrelated dependencies, or when nobody is watching the state transitions to tune it.