Streaming a Response Into the UI
The first version I shipped was synchronous. You typed a question, hit send, and the answer box sat empty for six, seven, sometimes ten seconds before the whole reply appeared at once. It worked. Every field was correct. And the support inbox filled with “the assistant is broken”, because a box that does nothing for eight seconds looks exactly like a bug.
We changed one thing. Not the model, not the prompt, not the words in the answer. We started showing each token the moment it was ready instead of waiting for the last one. The complaints stopped that week. Same time to the final word, give or take. A completely different product.
That is the whole subject here. Streaming does not make the model faster. It changes when the user first sees something, and that turns out to be the number they actually feel.
The model is already typing
A chat model does not compute the answer and hand it over. It generates one token at a time, feeding its own output back in to produce the next one, until it decides to stop. That is what a language model does at the mechanical level: an autoregressive loop, token after token.
So the tokens exist early. The first word is ready a few hundred milliseconds in, the second right after, and so on. When you make a plain synchronous call, the provider runs that entire loop, buffers every token, and only sends you the finished blob. You paid for a head start and then threw it away.
Streaming keeps the head start. Each token is sent the instant it is produced, so the first word reaches the screen almost immediately and the rest flows in behind it.
Total time to the final token is roughly the same in both. The bar you win on is time to first token, and that is the one perception hangs on.
The transport is a long-lived response
A synchronous call is one request and one response with the whole JSON body. Streaming needs something that stays open and drips out chunks over time. The usual answer is server-sent events: a normal HTTP response with Content-Type: text/event-stream that the server holds open and writes to as data becomes available.
The framing is deliberately dumb. Each event is a line that starts with data:, followed by a blank line. A streamed model response is a sequence of those, one per token or small group of tokens, and a sentinel at the end so the client knows generation finished:
HTTP/1.1 200 OK
Content-Type: text/event-stream
data: {"delta":"Stream"}
data: {"delta":"ing"}
data: {"delta":" shows"}
data: [DONE]Providers differ on the exact payload. One family sends a choices[0].delta.content fragment per event and closes with data: [DONE]. Another uses named events (content_block_delta and friends) and a structured stop event. The wire dialect changes. The shape does not: a long-lived response, framed chunks, a terminator. Learn to see that and any provider’s stream is legible.
Read it with fetch, not EventSource
There is a browser API named EventSource built for SSE, and it is the wrong tool here. Two reasons. It only does GET, but a model call is a POST carrying a messages array. And it cannot set request headers, so you cannot attach the auth your own endpoint needs. It is fine for a public firehose. It is useless for an authenticated chat POST.
So you read the stream by hand. A fetch response exposes its body as a readable stream, and you pull chunks off it as they arrive, the same progressive read covered in reading a response as it downloads. Decode the bytes to text, split on the blank-line frame boundary, and append each delta:
const res = await fetch("/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ messages }),
signal: controller.signal, // wired to a Stop button, see below
});
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += value;
const frames = buffer.split("\n\n");
buffer = frames.pop(); // keep the trailing partial frame for next time
for (const frame of frames) {
if (!frame.startsWith("data:")) continue;
const payload = frame.slice(5).trim();
if (payload === "[DONE]") return;
const { delta } = JSON.parse(payload);
if (delta) appendToBubble(delta);
}
}The one line people get wrong is buffer = frames.pop(). Chunks off the network do not line up with event boundaries. A single read() might hand you two and a half frames, and the next one carries the missing half. If you parse each read() in isolation you will JSON.parse a truncated data: line and throw. Keep a buffer, only process complete frames, and stash the leftover. That one habit removes a whole class of intermittent streaming bugs.
Your server relays the stream
The key never goes to the browser, so the browser never talks to the model directly. It talks to your server, and your server talks to the model. That rule from calling a model does not relax just because the response is now a stream. It means your server has to read one stream and write another, forwarding each chunk as it lands.
The simplest relay is a pass-through. In any runtime with web-standard streams, you can hand the upstream body straight back as your own response body:
export async function POST(req) {
const { messages } = await req.json();
const upstream = await fetch(MODEL_URL, {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.MODEL_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ model: MODEL, messages, stream: true }),
signal: req.signal, // forward the client's cancellation upstream
});
return new Response(upstream.body, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache, no-transform",
},
});
}That signal: req.signal line is doing quiet, important work, and we come back to it. When you need to do more than forward bytes (rename fields, count tokens for your own logs, enforce a per-user budget mid-stream, or strip the provider’s shape and emit your own), you pipe the upstream through a TransformStream instead of returning it raw. Same idea, one stage in the middle. Either way your server is the point where you can rate limit an abusive caller or cut a runaway generation off before it finishes.
Flush, or the whole thing is theatre
Streaming has one silent failure that wastes an afternoon the first time. Everything looks right in the code, and yet the browser sits blank and then dumps the full answer at once, exactly like the synchronous version you were trying to escape. Something in the middle buffered.
The usual culprits: a reverse proxy that buffers responses by default (nginx does, unless you turn proxy_buffering off or send the right headers), a framework helper that collects your writes and sends them on close, or gzip middleware that waits for enough bytes to compress. Any one of them re-assembles your careful little chunks into a single blob.
Two defenses. Send Cache-Control: no-transform so intermediaries leave the body alone, and make sure nothing in your own code accumulates the stream into a string before responding. The tell is blunt: your time to first token equals your time to last token. If those two numbers are the same, something buffered.
Deltas: text appends, tool JSON accumulates
Reading text deltas is easy. Each one is a fragment of the answer, so you concatenate it and show it. Done.
Tool calls are the trap. When a model streams a request to call one of your functions, the arguments arrive as a JSON string split into fragments, not as a finished object. You get {"ci, then ty":"Par, then is"}. Each piece on its own is not valid JSON, and if you try to JSON.parse every fragment you will throw on the first one, every time.
So handle the two delta kinds differently. Append text as it lands. For tool arguments, build up the string and only parse when the block closes:
let text = "";
let toolArgs = ""; // a STRING, grown fragment by fragment
for await (const event of stream) {
if (event.type === "text-delta") {
text += event.delta;
appendToBubble(event.delta); // safe to show right away
}
if (event.type === "tool-input-delta") {
toolArgs += event.delta; // do NOT parse yet
}
if (event.type === "tool-input-end") {
const args = JSON.parse(toolArgs); // now it is complete and valid
runTool(event.name, args);
}
}The event names vary by provider and by SDK. The rule underneath is stable: text is safe to display incrementally, structured arguments are not safe to use until the object is whole. This connects straight into tool calling and structured output, where those parsed arguments become real function calls you must still validate.
Stop means stop
A user reads the first two sentences, sees the answer heading the wrong way, and hits stop. If all you do is stop appending to the DOM, you have fooled the user and no one else. The model keeps generating on the provider’s side, and you keep paying for every output token nobody will ever read.
Cancelling for real means the signal travels the entire chain. The browser aborts the fetch, your server notices the connection closed, and your server aborts its own upstream call to the model. Only then does generation actually halt.
The client side is an AbortController. Create one per request, pass its signal to fetch, and call abort() when Stop is clicked:
const controller = new AbortController();
stopButton.onclick = () => controller.abort();
try {
const res = await fetch("/api/chat", {
method: "POST",
body: JSON.stringify({ messages }),
signal: controller.signal,
});
// ...read the stream...
} catch (err) {
if (err.name === "AbortError") return; // expected, not a failure
throw err;
}Aborting rejects the in-flight read with an AbortError, so wrap the loop in a try/catch and treat that name as a normal outcome, not a crash.
The server side is the signal: req.signal from the relay. In a web-standard runtime the incoming request carries an AbortSignal that fires when the browser disconnects, and forwarding it into the upstream fetch chains the whole thing together. Cancel at the top, and the cancellation flows down. On a classic Node server you get the same effect by listening for the request’s close event and calling abort() on your upstream controller. The mechanics of a connection going away mid-response are the same ones covered in graceful shutdown.
Connections die halfway
Streams break. A phone walks into a tunnel, a laptop sleeps, a load balancer times out an idle connection, a deploy cycles the server mid-response. The result is a stream that just ends, often with no terminator and no error, halfway through a sentence.
This is why the [DONE] sentinel and the stop reason matter. A clean end has a terminator or a finish reason attached. A dead connection does not. If your reader loop exits and you never saw either, you got a partial answer, and you should treat it as one: mark the message as incomplete rather than pretend it finished.
Resuming from the exact token where a stream died is possible but rarely worth it. You would have to persist partial state and replay it, and most chat UIs get more value from a plain “response interrupted, retry?” affordance. Retrying is a fresh request, not a resume, and the retry and backoff discipline you would use for any flaky upstream applies. One small thing that helps on the happy path: many servers send a periodic keepalive comment (an SSE line starting with :) during quiet stretches so idle-connection timeouts and buffering proxies do not sever a stream that is merely thinking.
Rendering without the flicker
Two rendering problems show up once tokens are landing continuously.
The first is partial markdown. If your answer is markdown and you re-parse the whole growing string on every token, a half-arrived code fence turns the entire document into a code block for a few frames, then snaps back when the closing fence lands. Half a link renders as literal brackets, then becomes a link. It flickers. The cheap fix is to render the streaming text as plain text and only run the markdown parser once, when the stream completes. If you want live formatting, use a parser that tolerates unclosed syntax, or only commit blocks that have visibly closed and keep the trailing incomplete line as plain text.
The second is speed. A model can push tokens faster than the browser can paint, and touching the DOM on every single delta is wasteful when several arrive within one frame. Coalesce them. Buffer incoming text and flush once per animation frame:
let pending = "";
let scheduled = false;
function enqueue(textDelta) {
pending += textDelta;
if (scheduled) return;
scheduled = true;
requestAnimationFrame(() => {
bubble.textContent += pending;
pending = "";
scheduled = false;
});
}Now a burst of ten tokens in one frame becomes one DOM write instead of ten, and the text still appears smooth because the eye only sees per-frame updates anyway.
This is a gentle cousin of a bigger topic. When the producer is genuinely faster than the consumer over a sustained period, buffering alone is not enough and you need real flow control, which is the whole story of backpressure. For rendering a chat answer, the per-frame flush above is almost always all you need.
Time to first token is the metric
If you measure one thing about a streaming feature, measure time to first token, separately from total time. Total time tells you how long the model took. Time to first token tells you how long the user stared at a blank box wondering if it broke, and that is the number that generates support tickets.
They move independently. A slow relay, a buffering proxy, or a heavy prompt that delays the first token all hurt time to first token without touching total time. Put both in your observability from day one, and alert on the first-token number. When someone says the assistant “feels slow”, this is almost always what they mean, and you cannot fix what you did not measure.
See it with no network
Here is a streaming renderer with no model and no network anywhere in it. A canned answer is tokenized in the page and fed into the bubble on a timer, with a blinking cursor while it types. Hit Stop to halt it mid-stream. Drag the speed slider to change the token rate, and watch how time to first token stays low even when the total time grows.
The point is the feel. Notice that the first word shows up almost the instant you press Start, no matter how slow you set the rest.
Slow it right down to 2 tokens a second. The total time balloons, but the first word still lands immediately, and the thing still feels alive rather than broken. That gap between the two numbers is the entire reason streaming exists.
Summary
- The model generates token by token anyway. Streaming sends each token as it is produced instead of buffering the whole answer, so the first word appears fast.
- The transport is a long-lived HTTP response, usually server-sent events: framed
data:chunks and a terminator. Provider dialects differ, the shape does not. - Read it on the client with
fetchand a readable stream, notEventSource, because a model call is an authenticatedPOST. Buffer across chunk boundaries and only parse complete frames. - Keep the topology browser to your server to the model. Your server holds the key and relays the stream, flushing every chunk. Buffering by a proxy, framework, or gzip silently collapses streaming back into a blob.
- Text deltas you append and show live. Tool-call arguments arrive as JSON string fragments you concatenate and parse once, never per fragment. See tool calling.
- Cancel for real. An
AbortControlleron the client, and forward the request’s signal to the upstream call on the server, so a Stop halts generation and stops billing, not just the DOM. - Connections die halfway with no terminator. Treat a stream that ends without
[DONE]or a finish reason as incomplete, and offer a retry rather than a fake-complete answer. - Render smooth: parse markdown once at the end (or with a tolerant parser), and coalesce token writes into one paint per frame with
requestAnimationFrame. - Measure time to first token separately from total time and put it in your observability. It is the number users actually feel.