Conversation State and Summarisation
A friend shipped a little party-planning assistant, the kind that helps you settle a menu over a long back-and-forth. On the second turn the user typed one sentence that mattered: “heads up, two guests are vegetarian and one has a peanut allergy.” The bot said the right things, they went deep on the plan, and about forty turns later, picking out snacks, it cheerfully suggested peanut satay skewers.
Nobody edited the allergy out. The user had just watched it scroll off the top of the conversation.
The app was doing something that sounds reasonable. To keep each request under the model’s limit, it kept only the most recent handful of messages and dropped the rest. Turn two was no longer recent. So the model never saw the allergy, because by then the allergy was not in the array it received. That is the whole subject of this page. The memory a chatbot appears to have is a thing you build by hand, and the way you build it decides whether a long chat stays coherent or quietly turns on the person using it.
The memory you feel is a list you resend
You met the core fact in messages and roles: a chat model is stateless. It keeps nothing between calls. What looks like memory is your own code resending the whole transcript every single turn, so that turn twenty arrives with turns one through nineteen stapled to the front of it.
That resend is the thing that costs you. Every previous message goes back up the wire on every call, which means a long chat gets slower and more expensive the longer it runs, and it eats into the context window, the fixed token budget that your system prompt, the history, any retrieved text and the reply all have to share. The transcript is the one part of that budget that grows without any ceiling of its own. Left alone, it walks straight off the edge.
So “keep the conversation going” and “keep the request under the limit” pull against each other, and something has to give. The rest of this page is the set of choices for what gives, from the crude one that bit the party bot to the one you actually want in production.
Three ways to keep it fitting
There are three answers you will reach for, and it is worth seeing all three next to each other before picking one, because the first two are traps that look fine in a demo.
Send everything
Do nothing, and the array just grows. This is fine for the first twenty exchanges and then it is not. A raw API call starts erroring with a “maximum context length” message once the input plus the reserved reply no longer fit, and a higher-level SDK often does something quieter and worse: it truncates the history for you, usually by dropping the oldest messages, which is exactly how a system prompt or an early fact vanishes with no warning. Either way you are also paying per token for a history that never stops growing. Sending everything is not a strategy, it is the absence of one.
Keep the last N turns
The obvious fix is a sliding window: before each call, throw away everything except the system prompt and the most recent N messages.
function lastN(system, history, n) {
// keep the standing rules, plus only the freshest N turns
const recent = history.slice(-n);
return [system, ...recent];
}This fits, it is trivial, and it will pass every test you write in an afternoon, because your tests are short. Then it ships into a real forty-turn conversation and does precisely what it says on the tin: it forgets. The name someone gave you on turn one, the constraint they set on turn two, the decision you reached together on turn five, all of it falls out the back of the window the moment it is no longer among the last N messages. The model is not confused. It is answering correctly given an array that no longer contains the fact. This was the peanut allergy. A sliding window does not know that turn two mattered more than turn thirty-nine; to it, older simply means gone.
Roll the old turns into a summary
The strategy that actually holds up keeps the shape of the sliding window but stops throwing the past away. Instead of deleting old turns, you compress them. Periodically you take the older part of the conversation, ask a model to boil it down to a short summary, and keep that summary pinned near the top. The most recent turns stay word-for-word. So the model gets two things at once: the gist of the entire conversation, and the full detail of the part that is still active.
That is the whole idea, and it is worth the rest of the page because doing it well is less obvious than it sounds.
How a rolling summary is put together
Picture the transcript split at a line. Everything to the left of the line, the old turns, gets folded into a single compact block of notes. Everything to the right, the recent turns, is kept exactly as it was. As the chat grows, the line slides forward: more turns cross into the summary, and the summary is rewritten to absorb them.
In code the loop is small. You keep the summary and the recent turns as separate pieces of state, and before each call you check whether the history has grown past a threshold. If it has, you fold the older turns into the summary and drop them from the verbatim list.
const KEEP_RECENT = 8; // most recent turns kept word-for-word
const TRIGGER = 0.7; // compress once we cross 70% of the budget
async function rollUp({ summary, history }, budget, count) {
const used = count(summary) + count(history);
if (used < budget * TRIGGER) {
return { summary, history }; // still roomy, do nothing
}
// fold everything except the freshest turns into the summary
const recent = history.slice(-KEEP_RECENT);
const older = history.slice(0, -KEEP_RECENT);
if (older.length === 0) return { summary, history };
const summary2 = await summarise(summary, older); // one cheap model call
return { summary: summary2, history: recent };
}Two numbers there are judgement calls, not laws. The trigger is how full you let the window get before compressing; squeezing at around 70 to 80 percent of the budget is a common starting point as of 2026, and you tune it, since summarising too eagerly wastes model calls and summarising too late risks an overflow before you get to it. KEEP_RECENT is how many turns stay verbatim; small enough to save tokens, large enough that the model still has the literal recent exchange to work with. Treat both as dials you turn against your own traffic, not defaults to trust.
The summarise call is itself a normal model request, and it is not free, so a rolling summary trades a bit of extra latency and cost now and then for a conversation that stays small and cheap on every turn in between. On a long chat that trade pays for itself quickly.
What to keep, and what to let go
A summary is only useful because it is shorter than what it replaces, and shorter means you are throwing information away on purpose. The whole skill is choosing what. Get specific in the prompt you hand the summariser, because “summarise this conversation” invites it to write a tidy paragraph that keeps the vibe and loses the facts.
The keep list is the useful part of the conversation stripped of its packaging:
- Decisions. Whatever the two of you settled on. “Menu is set on a taco bar” is worth ten turns of deliberation.
- Durable facts about the user. Their name, their constraints, their stated preferences. These almost never expire.
- Hard requirements. Budgets, deadlines, must-haves and must-nots. A stated limit is high-consequence and low-cost to keep.
- Open threads. What is still unresolved, so the model knows where to pick up rather than assuming everything is done.
The drop list is everything that was communication rather than content: hellos and thank-yous, context the user repeated because they forgot they had said it, and, importantly, superseded attempts. If you spent eight turns on a barbecue idea and then abandoned it for tacos, the barbecue detail is dead weight, and keeping it around actively confuses the model into thinking both plans are live.
A summariser prompt that spells this out beats a vague one by a wide margin:
const SUMMARY_PROMPT = `
You maintain a running summary of a conversation. Update it with
the new turns below. Keep it compact, in note form, preserving:
- decisions and choices the user has committed to
- durable facts about the user: name, constraints, preferences
- hard requirements: budgets, deadlines, must-haves, must-nots
- open threads: what is still unfinished
Drop greetings, thanks, restated context, and any approach that
was later replaced. Do not invent anything not in the turns.
Never drop a stated allergy, deadline, or spending limit.
`;When the summary drops the thing that mattered
Here is the part that keeps summarisation honest, and it is the reason the party bot is a cautionary tale and not just a bug. Compression is lossy by definition, and the model doing the compressing is guessing at what will matter later. It does not know that, twenty turns from now, the conversation will turn to snacks and the peanut allergy will suddenly be the single most important token in the transcript. To a summariser trying to be concise, one throwaway line about allergies buried in a long chat about themes and guest lists can read as minor. So it gets smoothed over, and then it is gone, and no error ever fires.
You cannot fully prompt your way out of this, because you are relying on a probabilistic model to correctly predict future relevance, which is not a thing it can reliably do. The fix is to stop relying on prose for the facts you cannot afford to lose.
Pull those facts out into a structured record, as data, and pin it into the context on every turn where the summariser can never touch it.
// extracted as data, kept separate from the prose summary,
// and re-sent verbatim on every turn so it can never be compressed away
const facts = {
name: "Dana",
dietary: ["vegetarian", "peanut allergy"],
event: { kind: "birthday dinner", guests: 10, date: "2026-08-14" },
budgetUsd: 200,
};Now the allergy is not a sentence competing for space in a paragraph the model rewrites every few turns. It is a field. It sits in its own layer, sent in full each time, and the summariser’s editorial choices cannot reach it. The prose summary can stay loose and human (“leaning toward a taco bar, still debating dessert”) while the load-bearing facts live somewhere with a schema.
One more habit that saves you later. The summary is a lossy view built for the model, so do not let it become your only copy of the conversation. Keep the full raw transcript in your own storage, a database row per message, as the source of truth. Then the summary is just a derived, disposable thing you can regenerate, tune, or throw away and rebuild if your compression prompt improves, without having lost a word of what actually happened.
Putting it together
Stack these ideas up and the context you send on a mature conversation has clear layers, each holding a different kind of memory, and each behaving differently as the chat goes on.
Assembling it is just building the array in that order:
function buildMessages({ system, facts, summary, recent, userTurn }) {
return [
{ role: "system", content: system },
{ role: "system", content: "Known facts (authoritative):\n" + JSON.stringify(facts, null, 2) },
summary ? { role: "system", content: "Conversation so far:\n" + summary } : null,
...recent, // the last N turns, untouched
{ role: "user", content: userTurn },
].filter(Boolean);
}Each layer earns its place for a different reason. The system prompt is your fixed steering. The facts are the non-negotiable specifics, pinned as data. The summary is the compressed past, cheap and approximate. The recent turns are the sharp, verbatim present. And you still reserve room for the reply, because the answer is generated into the same budget as everything above it.
That last durable-facts layer is the seam where this article hands off to the next one. Once you are extracting facts into a structured record, the obvious question is what happens when the conversation ends. Should Dana’s peanut allergy really be forgotten the moment she closes the tab, only to be re-learned next week? Persisting facts across sessions, and retrieving the right ones back into context when she returns, is long-term memory, and it is a genuinely separate problem from keeping one live chat inside the window.
All of this, deciding on each call what earns a place in the context and in what form, is context engineering applied to the one input that grows on its own. The conversation is just the first thing you learn to curate, because it is the first thing that overflows.
See the tradeoff bite
No model here and no network. Below is a canned ten-message conversation where turn two carries the fact that matters. The window only has room for the six most recent messages, so as the chat grew you had to make room. Pick how. Then ask the snack question and watch what the model can and cannot see.
Truncate, and the allergy is simply not in the array anymore, so the model does the reasonable thing with the information it has and suggests something with peanuts. Summarise, and the allergy is carried into a compact note that rides along at the top, so the same question lands safely. Same model, same question, different memory strategy, opposite outcome.
Nothing in that demo is smart. The toy summariser just copies a few known facts into a note, and the toy model just checks whether the word “peanut” is anywhere in the array it was handed. That is the honest version of the mechanism: the model can only use what is in front of it, and the entire game is making sure the right things are still in front of it when they suddenly matter.
Summary
- A chat model is stateless, so a chatbot only “remembers” because your code resends the transcript every turn. That transcript grows without bound and is the part of the context window most likely to overflow.
- Resending everything gets slow, gets expensive (you pay per token for the whole history each call), and eventually errors or gets silently truncated.
- A sliding window (keep the last N turns) fits, but it forgets on purpose: any fact older than N turns is gone, and it has no idea which old turn was load-bearing.
- A rolling summary keeps the recent turns verbatim and folds the older ones into a short summary that rides along at the top, so the model gets the gist of the whole chat plus the detail of the recent part. Compress at a threshold (around 70 to 80 percent of the budget is a common start), and keep updating the same summary rather than rebuilding it.
- Compress the right things: keep decisions, durable facts, hard constraints and open threads. Drop greetings, restated context, and attempts that were later replaced. Spell this out in the summariser’s prompt.
- Summarisation is lossy, and the model cannot predict which detail will matter later. For facts you cannot afford to lose (an allergy, a deadline, an account number), keep them as structured data pinned into every turn, not as prose the summariser can smooth away.
- Keep the full raw transcript in your own storage as the source of truth. The summary is a derived, disposable view built for the model.
- The mature setup layers it: fixed system rules, pinned structured facts, a rolling summary of old turns, recent turns verbatim, and reserved room for the reply. Persisting facts beyond a single session is long-term memory, and curating all of this per call is context engineering.