My LibraryBuy PDF
Part 6/Model Foundations

The Context Window as a Budget

A support agent we shipped had one hard rule in its system prompt: never promise a refund, always hand money questions to a human. It held that line for weeks. Then a customer had a long, winding chat about a delayed order, forty-something turns deep, and near the end the bot cheerfully offered them a full refund. Nobody changed the prompt. The rule was still right there in the code.

What happened is the whole subject of this article. The conversation had grown long enough that, to make room for the newest messages, the oldest ones got dropped, and the very first message in the list, the system prompt with the refund rule, was the oldest thing there. The model was not being defiant. It genuinely could no longer see its own instructions. They had fallen out of the window.

If you take one idea from this page, take this one. The context window is not “how much the model can read.” It is a single fixed budget that your system prompt, the entire conversation, any documents or tool results you stuff in, and the space left for the answer all have to share. They compete. Understand the competition and most of the weird behaviour you will hit stops being weird.

One window, one budget

You already know from tokens that the model reads a list of integers, not text, and that every operational limit is counted in tokens. The context window is the biggest of those limits: the maximum number of tokens the model can hold in mind for a single request. Input and output together.

Here is the mental model that fixes most beginner mistakes. Picture the window as a fixed-width bar. Everything you want the model to consider has to fit inside that bar, side by side, with nothing hanging off the edge.

One window, one budget. Four claimants compete for the same space.the model’s maximum context, a single fixed number of tokens2K48K18K8Ksystem promptfixedconversation historygrows every turnretrieved contextdocs, tool resultsroom to answerwhat is left overGive more to any one of them and the others get less.The room to answer is the slice that gets squeezed first.
The window is one bar with a fixed width. System prompt, history, retrieved context and the space for the answer all share it. Numbers are illustrative.

Four things share that bar, and it is worth naming them because people forget at least one every time:

  • The system prompt. Your instructions, the persona, the rules, the format you want back. It is usually pinned at the front and re-sent on every single call.
  • The conversation history. Every previous user message and every previous model reply. A chat model has no memory between calls; the only reason it “remembers” turn three when you send turn twenty is that turns one through nineteen are re-sent as tokens each time.
  • Retrieved context and tool results. Documents you pasted or looked up, the JSON a tool handed back, a chunk of a file. Anything you inject to ground the answer.
  • The space left for the answer. This is the one people forget. The reply is generated into the same window, so it needs room too, and it competes with everything above it.

What actually counts against it

The rule is blunt: everything you send plus everything the model generates draws from the same pool. Your whole input is read in one shot (the prefill), then the reply is produced one token at a time (the decode), and both live in the one window.

This is why the reply needs reserved space. When you call the API you tell it the most tokens the answer may use, often a parameter like max_tokens. The provider carves that space out of the window before generation starts. So the real ceiling on your input is not the whole window, it is the window minus the room you set aside for the reply.

const messages = [
  { role: "system", content: "You are Acme's support agent. Never promise a refund." },
  { role: "user", content: "hey, my order is late" },
  { role: "assistant", content: "Sorry about that. What is your order number?" },
  // ... 40 more turns pile up here ...
  { role: "user", content: "so what's the status?" },
];
// Every call re-sends this ENTIRE array. All of it counts against the window,
// and the reply still needs room on top.

When the budget blows

Push past the ceiling and one of two things happens, and they fail in very different ways.

Outcome A: a hard errorceilingovercontext_length_exceededthe whole request is rejected.you generate nothing, and pay nothing.Outcome B: silent truncationdroppedoldest messages dropped, no error raised.if the system prompt falls out, the modelforgets its own instructions.
Two ways a full window fails. A hard error rejects the whole request; silent truncation drops the oldest messages and the model carries on with a hole in its memory.

Outcome A, the hard error. You call the API, the request is too big, and it bounces with a 400-class error before a single token is generated. The message is some flavour of “maximum context length exceeded.” This is the honest failure. It is loud, it happens up front, and it forces you to deal with it.

{
  "error": {
    "type": "invalid_request_error",
    "message": "input length and max_tokens exceed context limit: 131200 > 128000"
  }
}

Outcome B, silent truncation. This is the one that bit our refund bot. A lot of chat frameworks and UI layers try to be helpful: when the history gets too long, they quietly trim it so the request still fits, usually by dropping the oldest messages. No error, no warning, the call succeeds. But the model is now reasoning over a conversation with the beginning torn off, and if your system prompt sat at the front, it is gone. The bot does not know it forgot anything. It just stops following rules it can no longer see.

Which one you get depends on your stack. Raw API calls tend to error. Higher-level SDKs and chat UIs tend to truncate. Knowing which behaviour your layer has is not optional, because the two demand completely different handling.

Here is the shape of doing it deliberately. Reserve room for the reply, always keep the system prompt, then fill the remaining budget with the newest turns and let the old ones go:

const WINDOW = 128_000;   // the model's ceiling, in tokens
const RESERVE = 4_000;    // room you refuse to give up for the reply

function fitContext(system, history, countTokens) {
  const budget = WINDOW - RESERVE - countTokens(system);
  const kept = [];
  let used = 0;
  // walk newest -> oldest and keep only what still fits
  for (let i = history.length - 1; i >= 0; i--) {
    const cost = countTokens(history[i]);
    if (used + cost > budget) break;
    kept.unshift(history[i]);
    used += cost;
  }
  return [system, ...kept]; // system prompt is never dropped
}

That is the crude version and it throws away old turns wholesale. Doing it well, summarising what you drop so the model keeps the gist, is its own topic in conversation memory. The point here is just that you want to be the one deciding, not a truncation step you never noticed.

Watch the budget fill up

Drag the sliders and watch the bar. Give the system prompt more room, add turns, add retrieved chunks, and the green “room to answer” shrinks. Push the total past the window and it goes red: at that point you are choosing between an error and dropping something. Flip the window size to feel how much headroom a bigger ceiling buys you.

interactiveA context budget simulator (fully simulated, no model)

Play with it for a minute and the tradeoffs get physical. A fat system prompt is a fixed tax on every turn. Retrieved chunks are expensive, a handful can eat more than a long chat. And a small window fills alarmingly fast, which is exactly why the industry spent the last two years making windows bigger.

Bigger windows raise the ceiling, not the problem

Windows have grown a lot. As of 2026 the frontier tier for hosted models is commonly around a million tokens, cheaper tiers sit near 128K, and a few open-weight models advertise far more. Those numbers keep climbing, so treat any specific figure as a snapshot rather than a fact to memorise. The direction is the durable part: up.

It is tempting to read that as “the problem is solved, just use the big one and paste everything in.” It is not solved. A bigger window moves two dials, and neither is the one you hoped.

A bigger window raises the ceiling and the bill. It is not a fix.128Kwindowsame content nearly fills it,little room, may truncate1Mwindowheadroom (empty costs nothing until you fill it)You pay per token you actually put in, and the bigger tier often costs more per token.Bars are schematic, not to scale: a 1M window is roughly eight times a 128K one.
A bigger window raises the ceiling and the bill. The same conversation leaves more headroom, but you still pay per token you put in, and the extra room is not free capability.

Dial one is headroom, and yes, you get more of it. Dial two is cost, and it moves the wrong way. You pay per token you actually send, so a bigger window does not make a big prompt cheaper; it just lets you build an even bigger, even more expensive one. The largest-window tiers frequently cost more per token on top of that. A generous window is permission to spend, not a discount.

And there is a third dial the marketing does not mention, which turns out to be the important one.

Why more context can make answers worse

A window has a number on it, but attention inside that window is not spread evenly. Feed a model a long context and it reliably pays most attention to the beginning and the end, and goes fuzzy in the middle. Put the one fact the answer depends on halfway through a long document and there is a real chance the model reads right past it.

The model does not read a long context evenly.recall of one buried factno-document baselineprimacy: strong at the startrecency: strong at the endfuzzy in the middlestart of contextmiddleend of contextPut the fact the answer depends on at the very start or the very end, not buried in the middle.
Recall of a single fact by where it sits in a long context. Strong at the start and the end, weak in the middle, sometimes worse than giving the model no document at all.

This is the “lost in the middle” effect, and it has been measured over and over since 2023. The curve is roughly U-shaped: high recall at the front, high at the back, a real sag in between. The unsettling part is that in the middle a model can do worse than if you had given it nothing at all, because a half-attended distractor is more misleading than an honest blank. It rhymes with how people recall a list, remembering the first and last items best, which is a strange thing to find in a machine whose attention mechanism can, in principle, reach any token equally.

It gets worse as the context grows, and not only at the edges of the window. Careful testing across many frontier models in 2025 showed accuracy dropping as input length increased well before the window was anywhere near full: a nominal 200K-token model losing ground at 50K, a million-token model that does not actually reason cleanly across a million tokens. People started calling it context rot. A few forces pile up at once. The lost-in-the-middle sag. Sheer dilution, because every extra token is one more thing competing for the model’s attention. And distractors, passages that look relevant but are not, which actively pull the answer off course.

So “just paste everything in” is not only expensive, it can lower your accuracy. The counterintuitive move is often the right one: send less, but send the right less. A short, sharply relevant context beats a giant kitchen-sink one on both cost and quality.

The real job is triage

Put it together and the engineering task comes into focus. You have a fixed budget, several things that want a place in it, and a strong reason to keep it lean rather than full. So the work is triage: on every call, decide what earns a seat.

The job: decide what earns a seat in the window.everything you could includefull chat historyevery retrieved documentall tool outputsthe whole system promptfew-shot examplescontextengineeringrelevant now?high signal?worth the tokens?what actually goes inrecent turns onlytop few doc chunkssystem prompt, trimmedroom to answerthe rest is summarised or left outMore is not better. What you leave out matters as much as what you keep.
On every call you choose what goes in the window. Curating that set down to what is relevant, recent and high-signal is the whole discipline of context engineering.

That triage has a name now, context engineering: the practice of curating, on each inference call, the smallest set of tokens that gets the job done. It is where a lot of the real work of building with models lives, and it pulls in most of the rest of this part. Retrieval decides which documents earn a place. Conversation memory decides how to compress a long chat so it keeps fitting. Watching your token cost tells you when a prompt has quietly bloated. The context window is the constraint that makes all of those worth doing.

None of this needs a bigger model or a cleverer prompt. It needs you to treat the window as what it is: a budget, with a hard limit, that you spend on purpose.

Summary

  • The context window is one shared budget measured in tokens. Your system prompt, the full conversation history, any retrieved documents or tool results, and the space for the reply all compete for the same fixed space.
  • Everything you send plus everything the model generates counts against it. The reply needs reserved room too, so your usable input is the window minus the space you set aside for the answer.
  • Overflow fails two ways. A raw API call errors with a “maximum context length” message before generating anything. Higher-level stacks often truncate silently, dropping the oldest messages, and if the system prompt falls out the model forgets its own rules with no warning.
  • Unbounded conversation history is the most common cause of these failures. Cap it on purpose: pin the system prompt, keep recent turns, summarise or drop the rest, rather than leaving it to a hidden truncation step.
  • Windows keep growing (roughly a million tokens at the frontier as of 2026, and rising), but a bigger window mostly raises the ceiling and the bill. You still pay per token you send, and the top tiers often cost more.
  • Attention is not uniform. Models recall facts best at the start and end of a long context and worst in the middle, and quality can degrade long before the window is full. More context can mean worse answers, not better.
  • The job is triage. Decide on every call what earns a place in the window. Sending less but more relevant context usually beats a giant one on both cost and accuracy, which is the whole point of context engineering.