One Thread, Many Requests

/healthz returns the string ok. No database, no disk, no allocation worth the name. One Tuesday it started timing out at three seconds, so the load balancer pulled that instance out of rotation, which pushed its traffic onto the other instances, which then also started failing /healthz.

CPU sat at 26%.

Twenty six percent. On a four-core box. If you have stared at that number long enough it is the whole tell: one core pegged out of four is 25%, and the dashboard averages across all of them. A single Node process can only ever peg one core with JavaScript. When it does, your CPU graph looks healthy and your service is face down.

The commit was one line, shipped that morning:

const hash = crypto.pbkdf2Sync(password, salt, 600_000, 32, 'sha256');

600,000 iterations is the current recommended work factor for PBKDF2-HMAC-SHA256. Being expensive is the entire point of it. On that box it took about 210ms.

pbkdf2Sync. Sync. On the thread. The only thread.

At six logins a second that one endpoint asked for 1,260ms of thread time out of every 1,000ms available. There is no version of that story where the queue drains. /healthz was never slow. /healthz was standing in a line that only got longer.

You already know the event loop from the browser side, where blocking it drops frames and jank is annoying. This is the same mechanism with a different blast radius. On a server, a blocked loop is not your problem. It is every user’s problem, including the ones on endpoints you didn’t touch.

The shape of a Node server

Start with the good case, because the good case is genuinely remarkable and it is why any of this is worth the trouble.

A typical handler does almost nothing. It parses a request, builds a query, and then spends the overwhelming majority of its life waiting on something that isn’t the CPU: Postgres, Redis, an upstream API, the disk. Measured honestly, a handler that takes 22ms of wall clock often uses about 1ms of thread.

So while it waits, the thread walks away and serves someone else.

Look at the bottom strip. That is the resource everyone worries about, and it is doing nothing for 17 of those 25 milliseconds while four requests are simultaneously alive.

Scale that up and the arithmetic gets silly in a good way. At 1ms of thread per request, one process has room for roughly 1,000 requests a second before the thread itself is the limit. What actually caps you first is memory, file descriptors, and your database connection pool. Not the thread. The thread is loafing.

Concurrency is not parallelism

The two words get swapped constantly, and the swap is exactly what makes people reach for the wrong tool.

Concurrency is how many things are in progress. Parallelism is how many things are executing right now.

The diagram above has a concurrency of four and a JavaScript parallelism of one. Four requests are alive; one line of JS is running. Node’s entire design is a bet that for a normal server, concurrency is the thing you need and parallelism is not, because your handler spends its life waiting for someone else’s answer.

That bet is excellent. It holds right up until a handler stops waiting and starts computing. Then it fails, and it fails for everyone at once.

What a blocked loop actually costs

Same picture. One change: the first handler hashes a password instead of querying a database.

The database is fine. The network is fine. /healthz is still four lines that return a string. Every one of those requests is slow because a different request is holding the thread, and JavaScript has no preemption. There is no scheduler that will interrupt pbkdf2Sync halfway through iteration 300,000 and let the health check squeak past. It runs to completion. Everyone waits.

This is head-of-line blocking, and the server-side version has a property the browser version doesn’t:

Your slowest handler sets the tail latency of every other endpoint.

Not the average. The tail. And you cannot fix /healthz by making /healthz faster, because /healthz was never the problem.

The arithmetic nobody does

Put a number on the thread. Call the fraction of wall-clock time it spends executing JavaScript its utilisation. Handlers using 1ms of thread each, at 400 requests a second, means the loop is busy 400ms out of every 1,000ms. Utilisation 0.4. Loads of headroom.

Now add a route that burns 200ms and gets three requests a second. That’s another 600ms per second. Utilisation 1.0, and the queue grows forever.

Here’s the part that surprises people. The trouble starts long before 1.0, because queueing delay doesn’t grow linearly with utilisation. It grows roughly like 1 / (1 - utilisation):

That’s the textbook M/M/1 result, and your traffic is not a textbook, so don’t take the numbers literally. Take the shape. It goes vertical at the end. That is why a server that has been “at 80% and totally fine” for six months falls over in an afternoon when marketing sends an email. You didn’t add 10% of load. You added 10% of the remaining distance to the wall.

It also explains why this never, ever shows up in dev. On your laptop, hitting the endpoint by hand, utilisation is roughly zero. There is no queue. The hash takes 210ms, the endpoint takes 210ms, you shrug and ship it. Queueing is a load phenomenon, and load is the one thing your laptop does not have.

Feel it

The demo below is the browser version of the same failure, because the browser lets you see the thread. A request arrives every 100ms and the handler does nothing but record how late it was. The clock and the spinner are driven by animation frames, which also need the thread.

One button does 1.2 seconds of work in one go. The other does the identical 1.2 seconds of work in 6ms slices, yielding in between. Watch the worst wait.

<div class="wrap">
<div class="head">
  <div class="spin" id="sc-spin"></div>
  <div class="clock" id="sc-clock">0.00s</div>
</div>
<div class="stats">
  <div class="stat"><span>requests served</span><b id="sc-served">0</b></div>
  <div class="stat"><span>worst wait</span><b id="sc-worst">0 ms</b></div>
  <div class="stat"><span>frames / sec</span><b id="sc-fps">--</b></div>
</div>
<div class="btns">
  <button id="sc-block">Hash it on the loop (1.2s)</button>
  <button id="sc-slice">Same work, 6ms slices</button>
</div>
<p class="note" id="sc-note">A request arrives every 100ms. Watch the clock, then watch the worst wait.</p>
</div>

* { box-sizing: border-box; }
.wrap { font: 14px/1.45 system-ui, sans-serif; padding: 14px; display: grid; gap: 12px; color: #171717; }
.head { display: flex; align-items: center; gap: 14px; }
.spin { width: 32px; height: 32px; border-radius: 50%; border: 3px solid #ececec; border-top-color: #7928ca; }
.clock { font: 700 26px/1 ui-monospace, monospace; min-width: 92px; }
.stats { display: flex; gap: 10px; flex-wrap: wrap; }
.stat { flex: 1; min-width: 108px; border: 1px solid #ebebeb; border-radius: 8px; padding: 8px 10px; display: grid; gap: 3px; }
.stat span { font-size: 11px; color: #777; }
.stat b { font: 700 17px/1 ui-monospace, monospace; }
#sc-worst { color: #d92d20; }
.btns { display: flex; gap: 10px; flex-wrap: wrap; }
button { font: 14px system-ui, sans-serif; padding: 8px 13px; border-radius: 8px; cursor: pointer; border: 1px solid #171717; background: #171717; color: #fff; }
#sc-slice { background: #fff; color: #171717; }
button:disabled { opacity: .45; cursor: default; }
.note { margin: 0; font-size: 12.5px; color: #666; min-height: 36px; }

const spin = document.getElementById("sc-spin");
const clock = document.getElementById("sc-clock");
const servedEl = document.getElementById("sc-served");
const worstEl = document.getElementById("sc-worst");
const fpsEl = document.getElementById("sc-fps");
const note = document.getElementById("sc-note");
const blockBtn = document.getElementById("sc-block");
const sliceBtn = document.getElementById("sc-slice");

const GAP = 100;    // a request arrives every 100ms
const WORK = 1200;  // total CPU work per click
const SLICE = 6;    // ms of work per slice

const t0 = performance.now();
let seq = 0;
let served = 0;
let worst = 0;

// --- simulated traffic: each request is its own timer with a known due time
function handle(due) {
const late = performance.now() - due;
served++;
if (late > worst) worst = late;
}

function armUpTo(horizon) {
const now = performance.now();
while (t0 + (seq + 1) * GAP < now + horizon) {
  seq++;
  const due = t0 + seq * GAP;
  setTimeout(() => handle(due), Math.max(0, due - performance.now()));
}
}
armUpTo(2500);
setInterval(() => armUpTo(2500), 500);

// --- a real yield: a task, not a microtask, so timers and painting get a turn
const chan = new MessageChannel();
const waiting = [];
chan.port1.onmessage = () => {
const resolve = waiting.shift();
if (resolve) resolve();
};
function yieldToLoop() {
return new Promise((resolve) => {
  waiting.push(resolve);
  chan.port2.postMessage(0);
});
}

function afterPaint() {
return new Promise((resolve) => {
  requestAnimationFrame(() => requestAnimationFrame(() => resolve()));
});
}

function burn(ms) {
const end = performance.now() + ms;
let acc = 0;
while (performance.now() < end) acc += Math.sqrt(acc + 1);
return acc;
}

async function sliced(total, slice) {
let done = 0;
while (done < total) {
  const chunk = Math.min(slice, total - done);
  burn(chunk);
  done += chunk;
  await yieldToLoop();
}
}

// --- the visible thread
let frames = 0;
let mark = performance.now();
function frame(now) {
frames++;
if (now - mark >= 500) {
  fpsEl.textContent = Math.round((frames * 1000) / (now - mark));
  frames = 0;
  mark = now;
}
clock.textContent = ((now - t0) / 1000).toFixed(2) + "s";
spin.style.transform = "rotate(" + ((now - t0) * 0.24) + "deg)";
servedEl.textContent = served;
worstEl.textContent = Math.round(worst) + " ms";
requestAnimationFrame(frame);
}
requestAnimationFrame(frame);

async function run(mode) {
blockBtn.disabled = true;
sliceBtn.disabled = true;
served = 0;
worst = 0;
note.textContent = mode === "block"
  ? "Working. The clock, the spinner and every queued request are frozen solid."
  : "Working. 6ms, yield, repeat. Everything else still gets a turn.";
await afterPaint();
if (mode === "block") burn(WORK);
else await sliced(WORK, SLICE);
note.textContent = mode === "block"
  ? "Done. Every request that arrived during those 1.2s waited for the whole thing."
  : "Done. Same CPU, a little more wall clock, but nobody else noticed.";
blockBtn.disabled = false;
sliceBtn.disabled = false;
}

blockBtn.addEventListener("click", () => run("block"));
sliceBtn.addEventListener("click", () => run("slice"));

Both buttons burn the same 1.2 seconds of CPU. The blocking one drives the worst wait to roughly 1,200ms, because a request that arrived 50ms in waited for the entire remaining job. The sliced one keeps it near the slice size.

Slicing is not free, and the demo is honest about that: yielding 200 times costs you some wall clock, and the frame rate still dips because each frame now competes with 6ms of work. That’s the trade. You pay a little of your own latency so that nobody else pays all of theirs.

Seeing it before your users do

Node will tell you, if you ask. monitorEventLoopDelay arms an internal timer every few milliseconds and records how late it actually fired. If the loop is free, that lateness is under a millisecond. If someone is hashing a password, it’s 200ms.

import { monitorEventLoopDelay } from 'node:perf_hooks';

const loop = monitorEventLoopDelay({ resolution: 20 });
loop.enable();

setInterval(() => {
  // the histogram records nanoseconds
  metrics.gauge('loop.lag.p50', loop.percentile(50) / 1e6);
  metrics.gauge('loop.lag.p99', loop.percentile(99) / 1e6);
  metrics.gauge('loop.lag.max', loop.max / 1e6);
  loop.reset();
}, 10_000).unref();

Three things worth knowing about that snippet. The histogram reports nanoseconds, so divide by a million or your dashboard will be nonsense. resolution is the sampling interval in milliseconds and defaults to 10, so you cannot detect a stall shorter than your resolution. And .unref() stops the interval from keeping the process alive at shutdown, which matters more than you’d think.

A healthy p99 loop lag is single-digit milliseconds. If yours is 50ms, then every request on that box, on every route, just inherited up to 50ms of latency it did nothing to earn. This is the single most useful number a Node process can emit and almost nobody exports it. Export it. Alert on it.

Getting the work off the loop

Here’s what fixing it looked like. Same traffic, same box, same day. The only change was moving the hash to a pool of worker threads.

Stare at that for a second, because it is the whole reason this topic matters.

The median is unchanged. If you were watching a mean-latency graph, or a p50 graph, you saw nothing happen on the day this broke and nothing happen on the day it was fixed. The damage lived entirely in the top 1%, and the top 1% is not a rounding error, it’s a slice of your users having a genuinely terrible time, plus the health check, plus whatever internal service was calling you with a 250ms timeout.

Both panels above share a y-scale, which is why the “after” peak is taller. Those aren’t new requests. They’re the same requests, come home from the tail.

So: how do you get the work off. There are five rungs on this ladder and you should climb them in order, because the cheap ones at the bottom solve more real problems than the interesting ones at the top.

Rung 0: don’t do the work

The fastest hash is the one you never compute.

Genuinely, check this first. A shocking amount of loop-blocking CPU is work being done at read time that could have been done once at write time, or work being recomputed per request that could sit in a cache, or a 40ms transform on a payload the client throws away. Rendering a PDF on every GET? Render it on POST and store it.

This rung is boring and it fixes more incidents than the other four combined. Password hashing, obviously, is not eligible. That’s the point of password hashing.

Rung 1: use the async built-in

Node ships async versions of most of the expensive things, and they don’t run on your thread. They run on the libuv thread pool.

import { pbkdf2 } from 'node:crypto';
import { promisify } from 'node:util';

const pbkdf2Async = promisify(pbkdf2);

// the loop is free for the whole 200ms
const hash = await pbkdf2Async(password, salt, 600_000, 32, 'sha256');

One word changed. The 200ms of SHA-256 now happens on a pool thread, the loop keeps serving /healthz, and you have fixed the outage.

You have not, however, made it free.

The pool defaults to four threads. It is shared, process-wide, by fs, by dns.lookup(), by zlib, and by the async crypto functions. UV_THREADPOOL_SIZE raises it (the ceiling is 1024), but read that sentence again before you go set it to 128: those threads still fight over the same physical cores. Raising the pool size on a 4-core box doesn’t add capacity, it just moves the queue somewhere you can’t see it.

Rung 2: worker threads

For CPU-heavy JavaScript that you wrote, there’s node:worker_threads. The docs are unusually blunt about the boundary, and they’re right: workers are for CPU-intensive JavaScript, and they do not help with I/O, where Node’s built-in async is already better than anything you’d build.

The rule that matters: use a pool. Spinning up a Worker means spinning up a fresh V8 isolate with its own heap, which costs tens of milliseconds and a chunk of memory. Do that per request and you’ve built something slower than the sync version you were trying to escape.

import { Worker } from 'node:worker_threads';

// one worker, reused, resource-limited. a real pool has N of these.
const worker = new Worker('./hash-worker.js', {
  resourceLimits: { maxOldGenerationSizeMb: 256 },
});

const pending = new Map();
let nextId = 0;

worker.on('message', ({ id, result, error }) => {
  const { resolve, reject } = pending.get(id);
  pending.delete(id);
  error ? reject(new Error(error)) : resolve(result);
});

function hash(password, salt) {
  const id = nextId++;
  return new Promise((resolve, reject) => {
    pending.set(id, { resolve, reject });
    worker.postMessage({ id, password, salt });
  });
}

Two details in there that people skip and then regret.

resourceLimits caps the worker’s heap. Without it, a runaway worker shares the process-wide heap limit and can take the whole process down with it, which rather defeats the isolation you came for. With it, you get an error event instead of a corpse.

And postMessage structured-clones its payload. For a password that’s nothing. For a 50MB buffer it’s a copy, on your thread, before the worker even wakes up, and you’ve quietly reintroduced the exact problem you’re solving. Use transferList to hand the buffer over instead of copying it, or reach for SharedArrayBuffer. The API here is a near-twin of web workers in the browser, so if you know one you know the other.

Before writing your own pool, look at piscina. Pooling, queueing, backpressure, cancellation and worker recycling are all fiddlier than they look, and it has done them already.

Rung 3: a queue

Now the question that outranks all of the above: does the caller actually need the answer right now?

Very often, no. Resizing an upload, generating the invoice PDF, reindexing the search document, sending the welcome email. The client wants to know you accepted the job, not that you finished it. So accept it, return 202 Accepted with an id, and do the work in a process whose whole job is doing work.

This is a better answer than a worker pool whenever it applies, because it decouples your web tier’s latency from the work entirely. A traffic spike lengthens the queue instead of your p99. That’s the trade you want. Background jobs and queues covers the machinery.

Rung 4: not JavaScript’s problem

Some work does not belong in your runtime at all. Video transcode, image pipelines at real volume, ML inference. Shell out, call a service, use the thing that’s good at it. Admitting this is not a defeat, it’s an afternoon saved.

Choosing a rung

Using the other cores

Everything so far kept one process. But one process runs JavaScript on one core, so a 16-core box running one Node process is renting 15 cores to nobody.

The fix is not more threads. It’s more processes.

cluster, and how it hands out connections

node:cluster is stable and has been for years. The primary forks N workers and they share a listening port:

import cluster from 'node:cluster';
import { availableParallelism } from 'node:os';

if (cluster.isPrimary) {
  for (let i = 0; i < availableParallelism(); i++) cluster.fork();

  cluster.on('exit', (worker, code, signal) => {
    console.error(`worker ${worker.process.pid} died (${signal || code})`);
    cluster.fork(); // see the warning below before shipping this line
  });
} else {
  startServer(); // each worker listens on the same port
}

They don’t really share the port. On Linux and macOS the default policy is SCHED_RR: the primary owns the listening socket, accepts every connection itself, and passes the file descriptor to a worker over IPC, round-robin. On Windows the default is SCHED_NONE, where the OS decides, and the docs are refreshingly honest that the distribution is bad in practice: something like 70% of connections landing on two workers out of eight. Override either with NODE_CLUSTER_SCHED_POLICY=rr or none.

Use availableParallelism() rather than os.cpus().length. It’s a thin wrapper over libuv, and on Linux it inspects the thread’s CPU affinity mask, so if you’ve been pinned to two CPUs it tells you two. It does not know about cgroup CPU quotas, though, which is exactly the thing containers use. On a pod limited to 2 CPUs on a 64-core node, you may well be told 64 and cheerfully fork 64 workers into a 2-CPU box. In a container, set the worker count from an env var and stop guessing.

Or let the kernel do it

Since Node v22.12.0 and v23.1.0 you can skip the primary entirely and have the kernel balance for you:

server.listen({ port: 8080, reusePort: true });

Every process binds the same port, and SO_REUSEPORT distributes incoming connections across the listening sockets. No primary, no IPC hop, no single accept loop to bottleneck on. For workloads with lots of short-lived connections it’s meaningfully faster than the round-robin dance.

One catch, and it will bite you on day one: it’s platform-specific (Linux 3.9+, FreeBSD 12+, DragonFlyBSD 3.6+, Solaris 11.4, AIX 7.2.5+) and it throws on platforms that don’t support it. Your macOS dev machine is one of those platforms. So:

server.listen({ port: 8080, reusePort: process.platform === 'linux' });

When to actually use cluster

Honestly? Less often than you’d think.

If you’re deployed anywhere that already runs replicas behind a load balancer (Kubernetes, ECS, Fly, Render, most PaaS), cluster builds a second scheduler inside the first one, and the second one is worse. Your platform’s memory limit is per-container, so N heaps in one container means one leaky worker starves its siblings and the OOM killer takes all of them. Your replica count is a number you can change in ten seconds; your worker count is a deploy. And the platform already restarts dead containers, with backoff, correctly.

Reach for it when you rent a big box by the month and want its cores, or you’re on bare metal or a single VM, or the baseline memory of N containers genuinely costs more than the complexity. That’s a real set of situations. It’s just smaller than the set of people using cluster.

What breaks the day you fork

This is the part nobody warns you about, so consider yourself warned. The moment you go from one process to N, every one of these becomes quietly wrong:

• That Map you used as a cache. There are four now. They disagree. A user updates their profile and sees the old name on three refreshes out of four.
• The in-memory rate limiter. 100 requests a minute times four processes is 400 requests a minute.
• The setInterval that sends the nightly digest. It sends four.
• The WebSocket connection registry. Your broadcast reaches a quarter of your users. See WebSocket for why the connection lives in exactly one process.
• Sessions in memory. Logged in on process 2, logged out on process 3, logged in again on process 2. Users describe this as “the site is drunk”.

None of these throw. None of them show up in tests, because your tests run one process. They just become wrong, and they get more wrong as you scale, which is the worst possible failure curve. The fix is always the same and it’s not clever: state that more than one process needs has to live somewhere both of them can see.

Things that block and don’t look like it

pbkdf2Sync is the easy one. It says “Sync” right on the tin. The ones that get you are the ones that don’t announce themselves.

JSON.parse and JSON.stringify. There is no async version and there isn’t going to be one. A 10MB request body is tens of milliseconds of solid thread time before your handler’s first line runs. This is the most common accidental blocker in real APIs, and the fix isn’t clever either: cap your body size. A body limit is not only a security setting, it’s a latency setting.

Server rendering. React’s renderToString is synchronous start to finish. A heavy page is 20 to 100ms of thread, per request, and renderToPipeableStream exists precisely because of that.

A regex over user input. A pattern with nested quantifiers can go exponential on a crafted string, which means an attacker gets to choose how long your event loop stops for. That’s catastrophic backtracking, and on a server it isn’t a slow request, it’s a denial of service against everyone on the box with a 30-byte payload.

Any *Sync in fs or zlib. readFileSync at boot is correct and good. readFileSync in a handler is a bug with the page cache papering over it, right up to the morning the file isn’t cached.

Loops over data you just fetched. SELECT 200,000 rows and .map() them and you’ve written a CPU-bound handler without typing the word “crypto” once. Same for .sort() on a big array, Buffer.concat on something huge, or building a 5MB string with +=.

The common thread: none of these have “Sync” in the name, and all of them hold the thread until they’re done.

And one that looks like it but isn’t

// 100 ids × 20ms each = 2 seconds
for (const id of ids) {
  const row = await db.query('select * from users where id = $1', [id]);
  rows.push(row);
}

Two full seconds. Awful. Fix it with Promise.all or, better, one query with where id = any($1).

But it is not blocking. The loop is free the entire two seconds, serving everyone else happily. This is a latency bug in one request, not a concurrency bug in the server. It hurts one user.

That distinction is worth internalising, because the fixes are unrelated and people reach for the wrong one constantly. await in a loop is slow. pbkdf2Sync is blocking. Slow hurts one person. Blocking hurts everybody. If you’re not sure which you have, the loop lag histogram from earlier will tell you in about ten seconds. See async/await if the difference between “waiting” and “working” is still fuzzy.

Summary

• One JS thread per process, in Node, Deno and Bun alike. Async I/O is what lets it serve thousands of connections, because a normal handler spends its life waiting, not computing.
• Concurrency is not parallelism. You get enormous concurrency and a JS parallelism of exactly one. Great trade for I/O-bound work, catastrophic for CPU-bound work.
• Blocking on a server doesn’t drop a frame, it stalls every other user on every other route. Your slowest handler sets the tail latency of everything else.
• Queueing delay grows like 1 / (1 - utilisation), so it goes vertical near the end. Invisible in dev, where utilisation is zero. It eats your p99 while your p50 and CPU graph look innocent.
• Export event loop lag from monitorEventLoopDelay (nanoseconds, so divide by 1e6) and alert on p99. Single digits is healthy. Then node --cpu-prof to find the frame.
• Move CPU work off the loop in order: don’t do it, use the async built-in, use a worker pool, use a queue, use something that isn’t JavaScript.
• The libuv pool behind those async built-ins is four threads by default, shared with fs, zlib and dns.lookup(). Saturate it and you starve DNS, which looks exactly like a slow database and isn’t one.
• Use the other cores with processes, not threads: cluster, reusePort on Linux, deno serve --parallel, or just N containers behind the balancer you already have. That last one is usually the right answer. And the day you fork, every in-memory cache, rate limiter and cron silently becomes wrong.