My LibraryBuy PDF
Part 5/Server Foundations

Reading a Request: Headers, Bodies and Why Parsing Is Hard

The ticket said “my name is wrong on my invoice”. The screenshot said Jos� Mart�nez.

We could not reproduce it. Typed the name into staging, saved, perfect. Typed it into production, perfect. Test suite green. The user tried again and it broke again, same two black diamonds, same two spots.

What cracked it was his bio field. Three kilobytes of Spanish, and it was mangled too, but only from roughly the 1400th character onward. Everything before that had perfect accents. Everything after was confetti.

The body had arrived in two pieces, and the split landed in the middle of a two-byte í. Our code did this:

let body = '';
req.on('data', (chunk) => { body += chunk; });  // the bug is on this line
req.on('end', () => {
  const data = JSON.parse(body);
});

body += chunk calls chunk.toString(). Once per chunk. Each half of that í got decoded on its own, neither half is a valid UTF-8 sequence, so each became U+FFFD, the replacement character. One broken letter, two black diamonds, and a bug that only appears when the payload is big enough to be split and the user is not named Bob.

That line is in a lot of tutorials. It is in a lot of production code. It is wrong.

The reason it is wrong is the one fact everything else here hangs off: req is not an object with your data on it. It is a stream.

The request is a stream, not an object

By the time your handler runs, Node has read and parsed the request line and the headers. It had to. The empty line that terminates the headers is the only marker of where they end, and Node cannot decide which handler to call until it has read the method and the path. So those come free:

req.method              // 'POST'          ready now
req.url                 // '/orders/42'    ready now
req.headers['content-type']  // 'application/json'   ready now

The body has not arrived. Depending on the client, it may not have been sent yet. Zero bytes of it are in memory. req.body does not exist, and no amount of squinting at the object in a debugger will make it appear.

What you actually have is a readable stream. IncomingMessage extends stream.Readable, so the request object is the same shape as a file read stream or a socket: it emits data events with Buffer chunks, then one end event, and you are responsible for deciding what to do with each piece as it lands.

chunks land one at a time and append to an array. nothing is parsed yet.your limit: 200 KBchunk 1+64 KBchunk 2+64 KBchunk 3+64 KBchunk 4crosses it64128192256 KBchunks 1 to 3: still under the limitkeep them, concat once at the endchunk 4: running total is now 256 KBstop counting, answer 413, drop the restthe running total is the only thing standing between you and an attacker with a big upload.nothing in Node adds it for you. there is no default body size limit in core.
Each data event hands you one Buffer. Nothing is parsed and nothing is validated. You decide whether to keep the bytes, and whether to keep counting.

Why body parsing is a middleware and not a property

This shape confuses people coming from PHP, where $_POST is just there. It looks like Node is being obtuse. It is not. If req.body were a property, something would have had to buffer the entire body before your code ran. For every request. Including the 200 MB video upload. Including the request you were about to reject with a 401 because the token expired. Including the one from a bot that opened a socket and declared it was going to send four gigabytes.

Reading the body is a decision, and only your code has the information to make it:

  • Auth middleware can reject a request without reading a single byte of the body.
  • An upload route wants to pipe the bytes straight to object storage and never hold them in memory at all.
  • A webhook route needs the exact raw bytes to check a signature, not a parsed object.
  • Your JSON route wants an object, and wants to give up at 100 KB.

Those are four different answers. A property can only give one. So the platform hands you the stream and lets the middleware layer decide, which is why express.json() is a function you call rather than something that is always on.

Reading a body by hand

Here is the whole thing, correct, in nine lines:

function readBody(req) {
  return new Promise((resolve, reject) => {
    const chunks = [];
    req.on('data', (chunk) => chunks.push(chunk));       // keep the Buffer as-is
    req.on('end', () => resolve(Buffer.concat(chunks))); // one decode point
    req.on('error', reject);
  });
}

The two highlighted lines are the fix for the invoice bug. Push the Buffer objects into an array untouched, join them with Buffer.concat at the end, and decode exactly once on the complete byte sequence. It matters because a UTF-8 character is one to four bytes and TCP has no idea what a character is. The chunk boundary falls wherever the network felt like putting it.

the string héllo, as the 6 UTF-8 bytes actually sent68c3a96c6c6fhé = 2 bytesllothe chunk boundary lands heretwo ways to turn those bytes into a string. only one survives.body += chunk (decodes each chunk alone)chunk 1 = 68 c3 gives h�chunk 2 = a9 6c 6c 6f gives �lloh��lloone letter in, two replacement chars outBuffer.concat(chunks).toString()all 6 bytes joined first68 c3 a9 6c 6c 6f decoded oncehéllothe é is still an é
TCP splits on byte boundaries, not character boundaries. Decode each chunk on its own and any character straddling the split is destroyed twice over.

The modern spelling is shorter, because IncomingMessage is async iterable:

async function readBody(req) {
  const chunks = [];
  for await (const chunk of req) chunks.push(chunk);
  return Buffer.concat(chunks);
}

Same semantics, no event wiring, and errors surface as a thrown exception you can try/catch like anything else in async/await.

Content-Type actually matters

You have a Buffer. It is meaningless until the client tells you what it represents, and the client tells you in exactly one place: the Content-Type header.

Three formats cover almost everything a browser will send you, and they are wildly different on the wire while meaning the same thing.

Content-Type: application/json{"name":"Ada","role":"admin"}Content-Type: application/x-www-form-urlencodedname=Ada&role=adminContent-Type: multipart/form-data; boundary=X--XContent-Disposition: form-data; name="name"(blank line)Ada--X(role repeats the same shape)--X--JSON.parseURLSearchParamsa real parserwhat your handler wanted{ name: "Ada", role: "admin" }multipart is the odd one out: the only format that can carry a file,and the only one you should never hand-roll yourself.
Three encodings of one identical pair of fields. The wire format differs completely; the object your handler wants does not.

application/json is the easy one. Decode to a string, JSON.parse it, wrap it in a try/catch because a malformed body is a 400 and not a crash. See JSON for the parsing rules themselves. application/x-www-form-urlencoded is what a plain <form> posts by default. It is a query string living in the body, so the parser is the same one you use for the URL:

const raw = (await readBody(req)).toString('utf8');
const fields = Object.fromEntries(new URLSearchParams(raw));
// 'name=Ada&role=admin'  ->  { name: 'Ada', role: 'admin' }

That Object.fromEntries has a sharp edge worth naming now: it keeps only the last value for a repeated key. a=1&a=2 becomes { a: '2' }, silently. If repeats are meaningful to you, use getAll instead. More on that below.

multipart/form-data is what a form posts when it has a file input, and it is a different animal. The body is a series of parts, each with its own headers, separated by a boundary string the client invented and declared in the Content-Type. Parsing it correctly means streaming, boundary matching across chunk edges, per-part headers, and filename handling. Do not write it yourself. Use a maintained parser (busboy, or multer which wraps it) and read file uploads, where the streaming-to-storage half of the problem lives. On the client side, FormData is what builds these bodies.

Path params, query, headers, body

Four places a client can put data, and picking the wrong one is a design mistake you live with for years.

the URL the client asked forhttps://api.shop.com/orders/42/items?tag=new&tag=saleschemehostpathquerysocket knowsgoes to Host headergoes in req.urlalso req.urlwhat the server actually reads off the socketPOST /orders/42/items?tag=new&tag=sale HTTP/1.1Host: api.shop.comAuthorization: Bearer ey...Content-Type: application/json(empty line){"qty":2}req.url = path + query onlyreq.headers.hostreq.headers.authorizationnot read yet. a stream.four places data lives, four different jobspath param42which resourcequery stringtag=newhow to shape the answerheaderAuthorizationmetadata about the requestbodyqty=2the payload you are sending
The URL gets taken apart and scattered across the message. req.url is only the path and query; the host arrives as a header; the body has not been read at all.

The rule of thumb that has never let me down:

  • Path params identify the resource. /orders/42. Remove it and you no longer know what you were talking about.
  • Query modifies the representation: filters, sorting, pagination. Remove it and you still get a valid answer, just a different slice of one.
  • Headers carry metadata about the message: who you are, what format you sent, what you accept, caching.
  • Body is the payload. The thing you are creating or changing.

Putting a filter in the body of a GET is technically possible and will be silently dropped by half the infrastructure between you and your server. Putting an auth token in the query string means it lands in access logs, browser history and Referer headers forever. Both are choices you make once and regret slowly.

req.url is not a URL

req.url gives you a path and query and nothing else. No scheme, no host. That surprises people every single time, and it follows directly from what goes on the wire: the request line only carries the target, and the host travels separately in the Host header.

So to use the URL object server-side, you have to reassemble it with a base:

const url = new URL(req.url, `http://${req.headers.host}`);

url.pathname;                    // '/orders/42/items'
url.searchParams.get('tag');     // 'new'    (the first one)
url.searchParams.getAll('tag');  // ['new', 'sale']

The plus sign that is a space

URLSearchParams implements the application/x-www-form-urlencoded rules, and those rules predate the modern percent-encoding everyone assumes. The one that bites: + decodes to a space.

new URLSearchParams('q=a+b').get('q');  // 'a b'   ← plus became a space
decodeURIComponent('a+b');              // 'a+b'   ← plus stayed a plus

Two decoders, same input, different answers, and both are correct for their own spec. If you hand-roll query parsing with decodeURIComponent and split('&') you will get this wrong, and the bug reports will be about search queries with spaces in them. A literal plus has to be sent as %2B. Have a play, and watch a query string come apart, repeats and all:

interactiveQuery string decoder

Try the “plus vs %2B” preset. q and r were typed almost identically and mean different things.

Headers are case-insensitive, and they repeat

Two facts that sound like trivia until they cost you a day.

Case is not significant. Node lowercases every incoming header name for you, so req.headers.host works no matter what the client sent. If you need the original casing (and you almost never do), req.rawHeaders is a flat array of [name, value, name, value, ...] with the bytes exactly as received.

A header can appear more than once, and what Node does about it depends entirely on which header it is. I sent a request with every header duplicated and printed what the handler saw:

sent on the wire req.headers gives you what happened
HOST: a.example then Host: b.example 'a.example' second one discarded
User-Agent: one then User-Agent: two 'one' second one discarded
Accept: text/html then accept: application/json 'text/html, application/json' joined with ,
X-Custom: p then X-Custom: q 'p, q' joined with ,
Cookie: a=1 then Cookie: b=2 'a=1; b=2' joined with ;
Set-Cookie (on responses) ['...', '...'] always an array

Three different behaviours in one object. The discard list is fixed and covers the headers where a second value is nonsense: host, authorization, content-length, content-type, user-agent, referer, from, etag, and a handful of others. Everything else joins with a comma. cookie joins with a semicolon because that is the cookie syntax. set-cookie is always an array because comma-joining cookies would corrupt them.

If you want the truth without the folding, Node gives you req.headersDistinct, where every value is an array:

req.headers.host;           // 'a.example'                 (lossy)
req.headersDistinct.host;   // ['a.example', 'b.example']  (all of it)

Charset, and the byte order mark

Content-Type: application/json; charset=utf-8 is on approximately every API request you will ever see. It is also meaningless.

JSON has exactly one encoding. The spec is blunt: “JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8”, and the media type registration adds that no charset parameter is defined for it, noting that adding one “really has no effect on compliant recipients”. So charset=utf-8 on JSON is a no-op that everyone sends anyway. Send it if you like, ignore it when you read it, and if someone sends charset=iso-8859-1 on JSON, they have a broken client and decoding as UTF-8 anyway is correct.

The same spec says implementations “MUST NOT add a byte order mark” to networked JSON. They do it anyway, because Excel exports, PowerShell’s Out-File, and a lot of .NET code add a UTF-8 BOM to everything they touch. And a BOM is fatal:

JSON.parse('{"a":1}');
// SyntaxError: Unexpected token '', ..."{"a":1}" is not valid JSON

Those three bytes (ef bb bf) survive Buffer.toString('utf8') as U+FEFF, land in front of the {, and JSON.parse refuses. Strip it before parsing if you accept uploads from the wild:

let text = buf.toString('utf8');
if (text.charCodeAt(0) === 0xfeff) text = text.slice(1);
const data = JSON.parse(text);

Urlencoded bodies are murkier. Percent-encoding produces bytes, and the format has no way to say which encoding those bytes are in, so a browser uses the encoding of the page the form was on. In practice everything has been UTF-8 for years and URLSearchParams assumes it, which is right often enough that you should not add configuration for it until something actually breaks.

The limit you have to set yourself

Here is the part beginners miss, and it is the one that pages you.

Node core has no default body size limit. Not “a generous one”. None. A bare http.createServer handler that buffers the body will buffer whatever it is sent, until the process runs out of heap and dies, taking every other in-flight request on that instance with it.

no limitlimit: 100 KBserver heapserver heapcapattacker streams a 5 GB bodyevery byte is appended to chunks[]heap exhausted, process killedevery other live request dies toothe same 5 GB bodythe running total trips at 100 KB413, then close the connectionmemory never movedone attacker, one socket, no botnet required. this is the cheapest denial of service there is.the limit belongs in your code, because Node core does not ship one.
The same 5 GB body against two servers. The difference is a running total and an early answer, not a bigger machine.

The check that looks right and does nothing

Reach for Content-Length and you get a limit that is free, instant, and fake:

// this looks like a size limit. it is not one.
const declared = Number(req.headers['content-length']);
if (declared > MAX_BYTES) {
  res.writeHead(413).end();
  return;
}

Two holes, and the second is fatal:

  1. Content-Length is a number the client typed. Trusting it to be honest is the whole problem restated.
  2. A chunked request has no Content-Length at all. Chunked transfer encoding exists precisely so a client can send a body without knowing its size up front, so the header is simply absent. Number(undefined) is NaN. NaN > MAX_BYTES is false. The guard waves it through.

I measured this rather than assuming it: the check above with MAX_BYTES at 1 KB, sent a chunked request with no Content-Length:

content-length header = undefined | transfer-encoding = "chunked"
ACTUALLY BUFFERED: 1048576 bytes
client saw: HTTP/1.1 200 OK

One megabyte through a one kilobyte limit, and a 200 on the way out. Keep the Content-Length check as a cheap early reject, because it saves you reading a body you were always going to refuse. It is just not a limit. The limit is the running total.

The version that works

function readBody(req, limit) {
  return new Promise((resolve, reject) => {
    const chunks = [];
    let size = 0;
    let done = false;

    req.on('data', (chunk) => {
      if (done) return;
      size += chunk.length;
      if (size > limit) {              // count the bytes you actually received
        done = true;
        const err = new Error('body too large');
        err.statusCode = 413;
        reject(err);
        return;                        // stop accumulating; later chunks hit the floor
      }
      chunks.push(chunk);
    });

    req.on('end', () => { if (!done) { done = true; resolve(Buffer.concat(chunks)); } });
    req.on('error', (err) => { if (!done) { done = true; reject(err); } });
  });
}

Once the total trips the limit, the done flag makes every later chunk a no-op. The bytes still arrive (you cannot un-send them) but they are never retained, so memory stays flat however long the attacker keeps typing.

Slow bodies, and the timeouts you inherit

Size is one axis. Time is the other, and it is the one people forget. An attacker does not need a big body to hurt you. They need a slow one. Open a connection, announce Content-Length: 10000000, then send one byte every 20 seconds. Your handler sits there with an open socket, an allocated buffer, and a promise that never settles. Repeat a few thousand times and you are out of sockets, having transferred almost no data. That is the slowloris family, and the variant that dribbles headers instead of a body works the same way.

The defence is timeouts, and Node ships defaults you should know rather than discover. Here they are as of Node 24 (the active LTS line in mid-2026), read straight off a fresh server object:

setting default what it bounds
server.headersTimeout 60000 (60s) time to receive the complete headers
server.requestTimeout 300000 (5 min) time to receive the entire request, headers and body
server.keepAliveTimeout 5000 (5s) idle time after a response before the socket closes
server.timeout 0 socket inactivity. zero means no timeout
http.maxHeaderSize 16384 (16 KB) total header bytes, past which you get a 431

Read that table twice. server.timeout is 0 by default, so there is no socket-level inactivity timeout at all. And requestTimeout gives every client five full minutes to finish sending a request. For an internal JSON API where a legitimate body arrives in under a second, five minutes is an eternity you are holding a socket open for. Tighten both to something that reflects your actual traffic:

const server = http.createServer(handler);

server.headersTimeout = 10_000;  // 10s to send me your headers
server.requestTimeout = 20_000;  // 20s to send me the whole thing
server.keepAliveTimeout = 5_000;

Set headersTimeout below requestTimeout, always. If headers alone are allowed to take as long as the whole request, the header phase can consume the entire budget and the split stops meaning anything.

So what does a body parser actually add?

Now the middleware makes sense, because you have hit every hazard it exists to cover. express.json() is not the thing that makes req.body work. It is a checklist:

  • A size limit. Default '100kb'. Exceed it and you get an error with status 413 and type 'entity.too.large', carrying limit and length so you can log what happened. Fastify does the same job with a bodyLimit option defaulting to 1048576 (1 MiB).
  • Content-Type matching, parameters and all, so application/json; charset=utf-8 matches and text/plain does not. Wrong type, no parse.
  • Charset handling, including bodies that genuinely are not UTF-8, with a 415 and type 'charset.unsupported' rather than mojibake.
  • Automatic inflate. inflate defaults to true, so a gzipped request body is decompressed. Note the ordering hazard: your limit has to apply to the decompressed size, or a zip bomb walks straight through a limit measured on the wire.
  • Correct concatenation and decoding. The invoice bug, handled, once, in a library thousands of people have already found the edges of.
  • strict mode. On by default, so only objects and arrays are accepted at the top level. JSON.parse('7') succeeds; express.json() rejects it, because a bare 7 as a request body is almost always a bug.

The urlencoded parser has its own defaults worth knowing: limit '100kb', parameterLimit 1000 (a cap on how many key/value pairs you will accept, which is a real DoS vector if left unbounded), and in Express 5 extended now defaults to false.

The option your webhooks depend on

The verify hook exists for a reason that catches teams out constantly. A webhook signature is an HMAC over the exact bytes the sender transmitted, so if your parser turns those bytes into an object and drops them, you cannot re-derive the signature: JSON.stringify(parsed) gives different bytes, because key order, whitespace and number formatting are all free to change. Keep the raw buffer while parsing:

app.use('/webhooks', express.json({
  limit: '1mb',
  verify: (req, res, buf) => { req.rawBody = buf; },   // stash the exact bytes
}));

Then verify the HMAC against req.rawBody, not req.body. Every payment provider’s integration guide has a version of this paragraph, and it is always the step people skip.

Summary

  • req is a stream, not an object. Headers are parsed and ready when your handler runs because Node needed them to route. The body has not arrived, which is why body parsing is a middleware you opt into and not a property.
  • Read a body by pushing Buffer chunks into an array and calling Buffer.concat once at the end. body += chunk decodes each chunk alone and shreds any multi-byte character that straddles a chunk boundary. for await (const chunk of req) is the modern spelling.
  • Content-Type decides the parser, and it carries parameters, so never compare it with ===. JSON is JSON.parse, urlencoded is URLSearchParams, multipart needs a real streaming parser and belongs with file uploads.
  • Four places data lives: path params identify the resource, query shapes the response, headers carry metadata, the body is the payload. req.url is only path plus query, and req.headers.host is attacker-controlled.
  • URLSearchParams decodes + to a space; decodeURIComponent does not. Use getAll when a key can repeat, because Object.fromEntries silently keeps only the last value.
  • Headers are lowercased and duplicates are folded three different ways: host and user-agent discard the second value, most headers join with , , cookie joins with ; . req.headersDistinct gives you the unfolded truth.
  • JSON is always UTF-8, charset on application/json is a no-op, and a leading BOM makes JSON.parse throw. Strip U+FEFF if you take bodies from the wild.
  • Set a body size limit yourself. Node core has none. Checking Content-Length alone is not a limit, because chunked requests do not send one. Count bytes as they arrive, answer 413, and do not req.destroy() before your response is out. requestTimeout defaults to 5 minutes and server.timeout to 0, so tighten those too.
  • A body parser is a checklist, not magic: limit, type matching, charset, inflate, correct concatenation, strict. Keep the raw bytes with verify when a signature depends on them.