Graceful Shutdown and Connection Draining
The 502s were folklore. Every deploy produced a small cluster of them, three or four seconds wide, thirty or forty errors, then the graph went flat again. Somebody had written it into the runbook two years earlier: brief 5xx during rollout, expected, no action required. Nobody had questioned it since, because the clients retried and the retries worked.
Then a customer was charged twice for the same 89 dollar order, and the folklore stopped being funny.
Here is the entire bug:
process.on('SIGTERM', () => process.exit(0));One line. It handles the signal, it exits cleanly, and it has a zero in it, which feels like success. It is worse than having no handler at all, because it looks like someone thought about shutdown.
At the moment that signal arrived, one of the pods was 44 milliseconds into a checkout:
app.post('/checkout', async (req, res) => {
const order = await db.query('INSERT INTO orders (...) VALUES (...) RETURNING id', [...]);
const charge = await payments.capture(order.total, req.body.token); // money moves here
await db.query('UPDATE orders SET status = $1, charge_id = $2 WHERE id = $3',
['paid', charge.id, order.id]); // never ran
res.json({ id: order.id });
});process.exit(0) does not unwind that. There is no rollback, no finally, no “hold on, I’m busy”. The heap ceases to exist between two lines of an async function. The card was charged. The row still said pending. The client got a connection reset, the mobile app retried the way it always did, and the retry captured a second charge against a card that had already paid.
Your process is not long-lived
You probably picture your server as something that starts up and then runs. Your orchestrator does not share that view. It kills your process on every deploy, every scale-down, every node drain, every rebalance, every spot reclaim, every time a health check gets grumpy. A service that ships ten times a day and autoscales through the evening gets terminated a few hundred times a month.
Shutdown is not an edge case. It is a code path you execute more often than most of your endpoints, and it is the only one where nobody notices you got it wrong, because the errors it produces look exactly like the errors a deploy is “supposed” to produce.
The default behaviour of that code path is to drop requests.
The two signals, and only one of them is a conversation
SIGTERM (signal 15) means please stop. It is a request. You can catch it, ignore it, or take your time answering it. That is the entire point of it.
SIGKILL (signal 9) is not a request. It cannot be caught, blocked, or handled, by you or by anyone. The kernel removes the process from the scheduler and reclaims its memory. No finally blocks run. No buffers flush. No log line gets written telling you it happened.
Everything that follows exists because you get the first signal some seconds before the second one, and that gap is the only time you will ever have.
Three numbers worth memorising, because you will read them off a dashboard at some point:
| Exit code | What happened |
|---|---|
0 |
You exited on purpose. Whether you finished your work is a separate question. |
143 |
128 + 15. SIGTERM ran to Node’s default handler. |
137 |
128 + 9. SIGKILL. You ran out of grace period, or the kernel OOM killer took you. |
A container that reports 137 on every single deploy is not a container with a slow shutdown. It is a container whose shutdown never happened.
How long is the gap? It depends on who is doing the killing, and you have almost certainly never set it:
- Kubernetes:
terminationGracePeriodSeconds, default 30 seconds. Measured from the start of the pod’s local shutdown, and it covers yourpreStophook and the process stopping. docker stop: 10 seconds by default, then SIGKILL. If you only ever test shutdown locally with Docker, you are testing against a third of your production budget.- Spot or preemptible instances: often less, sometimes much less, and the notice is not always a signal to your process.
The order is the whole lesson
Six things need to happen, and five of them are obvious. The interesting part is the order, because two of these steps look interchangeable and are not.
Read that list again and notice what is not first. Stopping the listener is step 3, not step 1. Everyone gets this backwards the first time, including me, and the reason is worth its own section.
Nobody has told the balancer about you
Here is the mental model that causes the bug: I got SIGTERM, so I should stop taking requests. It feels responsible. It is the same instinct as process.exit(0), just wearing better clothes.
The problem is that your load balancer has no idea any of this happened. It is a separate process, on a separate machine, working from a routing table it refreshes on its own schedule. At the instant SIGTERM lands, that table still lists you as a perfectly good backend, and it will keep handing you connections until something convinces it otherwise. If you slam the door on the first signal, every one of those connections gets refused, and that is your four-second cluster of 502s.
So the fix is to keep serving normally for a beat, while the news travels. Two places you can put that wait, and they are not equivalent:
In your handler, as a plain sleep before server.close(). Works anywhere, including on a bare VM behind an old-fashioned balancer. Costs you a few lines.
In a preStop hook, if you are on Kubernetes. This is better, and the reason is a detail most people miss: the kubelet runs preStop to completion before it sends SIGTERM. Your process does not even learn it is dying until the hook finishes, so it just keeps serving, at full readiness, on the code path you actually test. Meanwhile the control plane has been removing you from the EndpointSlice since the moment the pod was marked for deletion, and every routing layer downstream gets those seconds to catch up.
spec:
terminationGracePeriodSeconds: 30
containers:
- name: api
lifecycle:
preStop:
sleep:
seconds: 5The native sleep action has been enabled by default since Kubernetes 1.30, so you no longer need the old exec: ["/bin/sh", "-c", "sleep 5"] trick, which quietly required a shell to exist in your image. If you are on something older, or on a distroless image with no shell, that difference has probably already bitten you.
Readiness and liveness are not two names for the same probe
This trips up more teams than the signal handling does, because both probes are just an HTTP endpoint returning 200 and it is very tempting to point them at the same handler.
They have opposite consequences.
Which leads to the single most destructive thing you can put in a liveness probe: a check on a dependency you share with everyone else.
// This is a loaded gun pointed at your own cluster.
app.get('/healthz', async (req, res) => {
await db.query('SELECT 1');
res.send('ok');
});Think about what happens when the database has a four-second hiccup. Every pod fails liveness at once. The kubelet does the only thing it knows how to do and kills every one of them, simultaneously. Now your database, which was merely unhappy, is facing a full fleet of cold processes all reconnecting at the same moment, so it stays unhappy, so the new pods fail liveness too. You have converted a blip into an outage, and you did it with a feature designed to improve reliability.
Liveness should check almost nothing. “Is this process still able to answer?” is the whole question. If you cannot articulate what a restart would fix, it does not belong in there. A deadlocked event loop is a good liveness failure. A slow query is not.
let shuttingDown = false;
// Liveness: am I alive? Nothing else. No database, no Redis, no upstream.
app.get('/healthz', (req, res) => res.send('ok'));
// Readiness: should I get traffic? During drain, emphatically not.
app.get('/readyz', (req, res) => {
if (shuttingDown) return res.status(503).send('draining');
res.send('ok');
});Readiness on a shared dependency has a milder version of the same disease. The database blips, every pod goes unready together, and now the balancer has zero healthy backends. Some balancers fail open and route anyway. Plenty do not, and you have taken yourself down.
Closing the door properly
server.close() is the actual door. It does one thing and one thing only: it stops the listening socket from accepting new connections. It does not touch requests that are already running. Its callback fires later, once every connection has ended.
await new Promise((resolve, reject) => {
server.close((err) => (err ? reject(err) : resolve()));
});
// past this line: no new connections, and every in-flight request is doneThat promise is your drain. Everything already in the building gets to finish, nobody new gets in, and when it resolves you are genuinely idle.
Two things will stop it resolving.
Idle keep-alive connections. HTTP/1.1 clients hold their sockets open between requests. An idle socket is still a connection, so server.close() used to sit there waiting for a client that had no intention of ever speaking again. This is the origin of every “server.close() hangs forever” story you will find, and it is history: since Node 19.0.0, server.close() closes idle connections before returning. With Node 24 as the current Active LTS and 22 in its maintenance window, every release you should be running today has this. If you find server.closeIdleConnections() sprinkled through a codebase, that is a fossil from the Node 16 era and it is now a no-op you can delete.
Connections that are genuinely busy. A client streaming a slow upload, or waiting on a slow response, is not idle. server.close() will wait for it, correctly, and that is the point. But look at how long it is willing to wait:
| Setting | Default | What it means for you |
|---|---|---|
server.keepAliveTimeout |
5000 (5s) |
How long an idle socket lives. Irrelevant since Node 19 closes them for you. |
server.requestTimeout |
300000 (5 min) |
The ceiling on one request. Ten times your grace period. |
server.headersTimeout |
min(60000, requestTimeout) |
How long a client may dribble headers at you. |
server.timeout |
0 |
No inactivity timeout at all, since Node 13. |
requestTimeout is the one to stare at. Node is prepared to hold a single request open for five minutes. Kubernetes is prepared to wait 30 seconds for your entire process. One slow client, or one merely enthusiastic file upload, and your beautiful drain is still running when SIGKILL lands, taking every other in-flight request down with it.
So you need your own deadline, and it has to fire before the kernel’s.
const DRAIN_DEADLINE_MS = 20_000;
async function drain() {
const done = new Promise((resolve) => server.close(resolve));
const timer = setTimeout(() => {
log.error({ open: server.connections }, 'drain deadline hit, forcing sockets closed');
server.closeAllConnections(); // in-flight requests die here, deliberately
}, DRAIN_DEADLINE_MS);
timer.unref(); // never let this timer be the reason we stay alive
await done;
clearTimeout(timer);
}server.closeAllConnections() (Node 18.2+) is the hammer. It destroys every established connection including the busy ones. The docs recommend calling it after server.close() rather than before, to avoid racing the listener teardown, which is exactly the shape above.
The unref() matters more than it looks. Without it, a 20 second timer is itself a live handle on the event loop, so a process that drained cleanly in 3 seconds would sit around for another 17 waiting for a timer whose only job was to kill it.
Play with the drain
Below is the whole thing in miniature. Traffic arrives on its own. Hit SIGTERM and watch two separate things happen: new arrivals start getting refused (amber, and this is correct behaviour, they are the propagation gap made visible), while requests that already started keep running to completion.
Then drag the grace period down to 2 or 3 seconds and hit SIGTERM again. That is what a budget that is too short looks like from the inside.
Notice what the amber bars are telling you. They are not errors in the simulator. They are the requests the balancer sent you after you decided to stop taking them, and the only way to make them disappear is to wait longer before you close the door.
The front door is not the only door
server.close() closes the listener. If your process also pulls work from a queue, none of this touches it.
Picture the sequence. SIGTERM arrives, you start your careful HTTP drain, and for the next eight seconds your queue consumer, which has never heard of any of this, cheerfully long-polls, grabs ten more jobs, and starts them. At second 30 the kernel kills you and those ten jobs die halfway through. You have written an elegant shutdown for one half of your traffic and pretended the other half does not exist.
The shape of the fix is identical to HTTP, and so is the order: stop pulling, then wait for what is running.
async function shutdown() {
shuttingDown = true;
await sleep(PROPAGATION_MS);
await consumer.stop(); // stop *fetching*. in-flight jobs keep going.
await drain(); // HTTP in-flight
await consumer.drained(); // now wait for the jobs that already started
}Whatever your library calls it (worker.close(), consumer.stop(), subscription.close()), the contract you want is that one, and it is worth reading the source to check you are getting it. Plenty of clients have a single close() that means “stop fetching and abandon what is running”, which is not the same thing at all.
The stakes here are higher than with HTTP, and here is why. A job pulled off a queue and killed mid-execution does not politely disappear. Most brokers work on a visibility timeout: the job is hidden while you hold it, and if you never acknowledge it, it comes back. Which is the correct behaviour, and it means a half-finished job is a job that will run again from the top. If it had already charged a card, sent an email, or incremented a counter, it will do that a second time.
Two more things that pull work in and will happily outlive your drain:
setIntervaltimers. The metrics flush, the cache refresher, the nightly digest. Beyond the obvious problem, an interval is a live handle on the event loop, so a process that never callsclearIntervalwill never exit on its own even after everything else is done. You will watch it sit at idle for the full grace period and then get killed, and the logs will show nothing, because there is nothing to show.- In-process schedulers. Anything holding a
setTimeoutfor future work is holding both a job you are about to lose and a reason your process will not exit.
The arithmetic, which is where most of this dies
Every timeout you have just added has to fit inside a budget you did not choose:
propagation wait + drain deadline + resource close < terminationGracePeriodSecondsStrictly less than. With margin. And remember the preStop hook comes out of the same 30 seconds, because Kubernetes is explicit that the grace period covers the hook and the container stopping.
Pick the numbers from your actual latency, not from a blog post. If your p99 request is 800ms, a 20 second drain is absurdly generous and you will never touch it. If you have a report endpoint that runs for three minutes, no grace period will save you and you do not have a shutdown problem, you have a design problem: that work belongs on a queue. Long requests and frequent deploys are fundamentally at war, and the requests lose.
The signal that never arrives
Everything above assumes SIGTERM reaches your Node process. Sometimes it just doesn’t, and the symptom is beautifully specific.
If your container takes exactly the full grace period to stop, every single time, your handler is not running.
Not “sometimes takes a while”. Exactly 30.0 seconds, or exactly 10.0 under docker stop. Real drains are messy and take 2.3 seconds or 6.8 seconds. A number that lands precisely on the timeout is not a slow shutdown, it is a shutdown that never started, and the process was sitting there ignoring a signal it never received.
The cause is nearly always PID 1:
CMD npm start # npm spawns node as a child. Your handler never fires.
CMD node server.js # shell form: /bin/sh -c "node server.js". sh is PID 1. sh does not forward.
CMD ["node", "server.js"] # exec form: node IS PID 1. The signal lands where you want it.Shell form and npm start both put something between the kernel and your process that has no interest in passing signals along. Use exec form. If your image genuinely needs to spawn child processes, add a real init (docker run --init, or tini) so somebody is reaping zombies and forwarding signals properly.
The test takes twenty seconds and you should run it today:
docker run -d --name t my-api
time docker stop t
# 0.3s → your handler ran. good.
# 10.0s → your handler did not run at all. fix your CMD.Same idea in a cluster: kubectl delete pod one instance and watch the clock. See deploying apps for where this sits in the wider release story, and CI/CD for making the check automatic.
Putting it together
const DRAIN_DEADLINE_MS = 20_000;
const PROPAGATION_MS = 5_000; // skip this if a preStop hook does the waiting
let shuttingDown = false;
async function shutdown(signal) {
if (shuttingDown) return; // a second SIGTERM must not restart the dance
shuttingDown = true;
log.warn({ signal }, 'shutdown: started');
// 1. readyz is already returning 503, because the flag above flipped.
// 2. keep serving while that news reaches every routing table.
await sleep(PROPAGATION_MS);
// 3. stop pulling work in, both doors at once
clearInterval(metricsTimer);
await consumer.stop();
// 4. stop accepting connections and let in-flight requests finish
const closed = new Promise((resolve) => server.close(resolve));
const forceAt = setTimeout(() => {
log.error({ open: server.connections }, 'shutdown: deadline hit, forcing');
server.closeAllConnections();
}, DRAIN_DEADLINE_MS);
forceAt.unref();
await closed;
clearTimeout(forceAt);
// 5. nothing can start new work now, so it is safe to take the tools away
await consumer.drained();
await pool.end();
await redis.quit();
log.warn('shutdown: complete');
}
for (const signal of ['SIGTERM', 'SIGINT']) {
process.on(signal, () => {
shutdown(signal).then(
() => process.exit(0),
(err) => { log.error({ err }, 'shutdown: failed'); process.exit(1); },
);
});
}The bit that is easy to get wrong is step 5’s position. Close the connection pool while requests are still draining and every one of them fails on a dead pool, which is a fresh way to lose the exact requests you were trying to protect. Release resources last. They are what the in-flight work is using.
The if (shuttingDown) return guard matters too. Impatient humans send a second SIGTERM. Some orchestrators repeat it. Without the guard you get two overlapping shutdowns racing to close the same pool, and the second one throws inside a handler that has nowhere to report it.
Zero downtime is just overlap
All of this pays off in one place: the rollout. A deploy is not a restart, it is a replacement, and the only property that makes it invisible is that the new instance is taking traffic before the old one stops.
The dashed green line is the whole trick. Everything before it is the new instance booting and failing its readiness check on purpose, which is the same readiness endpoint doing the same job it does during shutdown, just in the other direction. An honest readiness probe is what buys you both halves.
Two ways to lose the overlap and both are common. Set maxUnavailable above zero and you have explicitly told the orchestrator that a dip is acceptable, so it will take one. Or make your readiness probe return 200 the instant the process starts, before the connection pool is warm and the config is loaded, in which case the overlap exists on paper and your first hundred requests hit a cold instance that lies about being ready. A readiness probe that always says yes is a readiness probe you do not have.
It will still not be enough
Do all of this correctly and you will still lose requests. Not many. But the rate is not zero, and it never will be.
SIGKILL cannot be handled. The kernel’s OOM killer sends it without warning or notice. A node can lose power. A kernel can panic. A hypervisor can vanish. A spot instance can be reclaimed faster than its notice period promised. In every one of those cases, in-flight work is destroyed and no code you write runs.
Which means graceful shutdown is a frequency reduction, not a fix. It takes a problem that happens forty times per deploy and makes it happen roughly never, which is enormous and worth every line above. But the client is still going to retry sometimes, and the only thing that makes a retried request safe is that doing it twice is the same as doing it once.
Go back to the checkout from the opening. A proper drain would have saved that customer, that day. What would have saved every customer, on every day, including the day the node caught fire, is that the retry should have been a no-op against a charge that already existed. Both. Not either. That is idempotency keys, and it is the natural next thing to read.
The right way to hold this: shutdown handling makes your deploys quiet. Idempotency makes your system correct. Doing the first one well is what earns you the right to only occasionally need the second.
Summary
- Your process is killed constantly (deploys, autoscaling, node drains, spot reclaims), so shutdown is a hot code path, not an edge case. Its default behaviour is to drop requests.
- SIGTERM is a request you can catch. SIGKILL is not, ever. Exit code 143 means SIGTERM was handled; 137 on every deploy means your shutdown never ran.
- Installing
process.on('SIGTERM')removes Node’s default exit. Now you own exiting, and both “exits too early” and “never exits” come from that one line. - Order: flip readiness to 503 → keep serving while the news propagates →
server.close()→ drain in-flight → stop consumers and close the pool → exit. Stopping the listener first is the classic bug, because the balancer has not heard about you yet. - The propagation wait belongs in a
preStophook if you have one (Kubernetes runs it before SIGTERM, and the nativesleepaction is default-on since 1.30). Measure the duration, don’t copy it. - Readiness removes you from the pool. Liveness gets you killed. Never put a shared dependency in a liveness probe unless you want one database blip to restart your entire fleet at once.
- Since Node 19,
server.close()reaps idle keep-alive connections for you. Busy ones it waits for, andrequestTimeoutdefaults to five minutes against a 30 second grace period, so you need your own deadline pluscloseAllConnections()as the hammer.unref()that timer. - The queue is traffic too. Stop pulling before you wait for what is running, or you will be killed holding jobs that a visibility timeout will hand to someone else to run a second time.
wait + drain deadline + close < terminationGracePeriodSeconds, strictly, with margin. A drain timeout equal to the grace period never fires.- If your container takes exactly the grace period to stop, the signal isn’t reaching Node. Use exec-form
CMD ["node", "server.js"], notnpm startor shell form. - WebSockets and SSE never drain on their own and
closeAllConnections()skips upgraded sockets. Close them yourself, early. - None of this reaches zero. Pair it with idempotency keys, because the retry is coming either way.