My LibraryBuy PDF
Part 6/Memory and Context

Long-Term Memory Across Sessions

A teammate described our internal assistant as “talking to someone with a great memory of the last ten minutes and no memory of anything before that.” Every Monday she opened it and typed the same three sentences: the service she owned, the database it ran on, and that she wanted examples in TypeScript, not Python. Once it was caught up it was genuinely useful. Catching it up was a tax she paid every single time, and after about two weeks she stopped paying it and went back to searching the wiki.

The model was not the problem. It was doing exactly what a stateless model does.

You saw in conversation memory that a chatbot only feels like it remembers because your code resends the transcript on every turn. That trick keeps one chat coherent. It also ends the instant the tab closes. The transcript is gone, and with it everything the assistant “knew” about her. Long-term memory is the thing that lets Monday’s session already know she is on Postgres and wants TypeScript, without her saying it again.

This sounds like a model capability you switch on. It is not. It is a retrieval system you build, and it is the same RAG loop you already know, pointed at facts about a person instead of documents. You persist the facts worth keeping, and on a later turn you pull back the relevant ones and paste them into the prompt. Storing is the easy ten percent. The failures, and there are several distinctive ones, live in deciding what to keep, keeping it current, retrieving the right subset, and not turning a helpful feature into a privacy incident.

One store is the whole difference between amnesia and continuity.WITHOUT long-term memoryMonuser: I use Postgres,answers in TypeScriptassistant: got itTueuser: I use Postgres,answers in TypeScript(typed it again)Weduser: I use Postgres,answers in TypeScript(and again)Close the tab and it is gone. Every session starts blank, so she repeats herself.WITH long-term memoryMonstates it once:Postgres, TypeScriptwritememory storePostgres, TypeScriptTue: knows, straight to itWed: knows, no re-introState it once, it is waiting next week. That is the whole feature, and the whole risk.read path is cheap: pull the few relevant facts, paste them in, answer.
The same three sessions, with and without a place to store facts. Without one, every session restarts blank and the user re-types the same context. With one, she states it once and it is waiting next week.

It is retrieval wearing a friendly name

Hold the two kinds of memory next to each other, because people blur them and they are different jobs. Conversation memory works within one session: it keeps the current chat from overflowing the window by resending and summarising the transcript. Long-term memory works across sessions: it survives the chat ending, and its job is to bring the right facts back the next time this specific person shows up.

The mechanism is retrieval, top to bottom. There is a store. Something writes durable facts into it. Later, on a new turn, you read the relevant ones out and inject them into the context, exactly the way RAG pastes retrieved document chunks into a prompt. Swap “document chunks” for “facts about the user” and almost everything you learned about retrieval carries straight over: you embed, you rank by relevance, you keep a tight budget, and the whole thing is only as good as what comes back.

The loop: extract, store, retrieve

Three moves make up the whole system. Two of them run when a conversation happens; one runs at the start of a later turn.

Extract. After (or during) a conversation, a small, cheap model call reads the recent turns and pulls out the durable facts as data. This is a separate call from the one talking to the user, and its only job is to turn “actually, we moved off MySQL, everything is Postgres now” into a stored preference. Ask it for structured output so you get parseable records, not a paragraph.

// A small, cheap model call, separate from the chat. Its whole job is to
// read the latest turns and hand back durable facts as data, not prose.
const facts = await extract({
  system:
    "Extract durable facts about the user worth remembering next session: " +
    "stable preferences, projects, decisions, hard constraints. " +
    "Ignore greetings, one-off questions, and anything transient. " +
    "Return an empty list if nothing is worth keeping.",
  input: recentTurns,
  schema: MemoryList, // e.g. array of { kind, content, confidence }
});

Store. Each fact becomes a row keyed to its owner, with enough metadata to manage it later: who it belongs to, what kind of fact it is, where you learned it, and when. Provenance matters more than it looks. When a memory turns out to be wrong, source is how you trace it back to the turn that created it.

const memory = {
  id: "mem_8f2",
  userId: "u_412",          // the owner. every read filters on this.
  kind: "preference",       // preference | project | decision | fact
  content: "Uses Postgres, not MySQL",
  source: "conv_119#turn7", // provenance: where you learned it
  createdAt: "2026-07-02T09:14:00Z",
  updatedAt: "2026-07-02T09:14:00Z",
  supersededAt: null,       // set when a newer fact replaces this one
};

Retrieve. On a new turn, you find the memories relevant to what the user is doing right now and inject them. The read is a scoped query: filter to this user, skip anything a newer fact has replaced, and order by distance to the current context. If you store an embedding per memory it is the same nearest-neighbour search you use for documents, and the same pgvector <=> operator.

-- Retrieve this user's memories most relevant to the current context.
-- Scoped to the caller: a memory belongs to exactly one user, always.
SELECT id, kind, content, updated_at
FROM memories
WHERE user_id = $1
  AND superseded_at IS NULL   -- skip facts a newer one replaced
ORDER BY embedding <=> $2      -- cosine distance to the current context
LIMIT $3;

The one structural difference from document RAG is the timing of the write. In RAG, ingestion is a deliberate offline job you run when a document changes. Here, the “documents” are produced by the conversations themselves, continuously, as a side effect of people talking to your assistant. So the write path runs after turns, and you almost never want it on the hot path. Extraction is a second model call; make the user wait for it and every reply gets slower. Push it into a background job so the user gets their answer immediately and the memory is written a beat later.

one loop, two paths, one storewrite path: runs after a conversation, off the hot pathconversationrecent turnsextractsmall model callreconcileadd, update, skipwritesMEMORY STORE (per user)one row per fact: content, kind, embedding, provenance, timestampsreads top-kread path: runs at the start of a later turnnew user turnplus contextretrieve, injectinto the promptmodel answers
Long-term memory is one loop on two paths that meet at the store. The write path extracts durable facts from a conversation off the hot path. The read path pulls the relevant few back into a later prompt.

Injecting the retrieved memories is the same string assembly you did for RAG. Put them in a system message so the model treats them as standing context, then the recent turns, then the new question.

function buildMessages({ system, memories, recent, userTurn }) {
  const known = memories.map((m) => `- ${m.content}`).join("\n");
  return [
    { role: "system", content: system },
    memories.length
      ? { role: "system", content: `What you remember about this user:\n${known}` }
      : null,
    ...recent,
    { role: "user", content: userTurn },
  ].filter(Boolean);
}

Nothing here is exotic. It is the RAG pipeline with the corpus swapped and the write path moved into the background. Which is exactly why it is worth building yourself once: the parts that make it feel like magic are parts you already understand.

What is worth remembering

Here is where the toy and the product diverge, and it is not the code. It is judgement about what to store.

Get greedy and remember everything, and two things go wrong. Retrieval gets noisier, because now the store is full of “asked about pricing once” and “said thanks” competing with the facts that matter. And you carry stale trivia forever, so the assistant confidently brings up a preference the user mentioned offhand a year ago and has since changed their mind about. Roughly two thirds of any conversation is small talk, restated context, and transient reasoning that is worthless next week. Storing it is negative value.

Remember too little and you are back to the amnesiac assistant, which defeats the point.

So extraction is selective on purpose. The prompt does the work: you are asking the model to keep the durable, high-signal facts and drop the rest. In practice the keep list looks a lot like the one from summarisation, because it is the same instinct applied to a longer horizon.

  • Stable preferences. How they like answers, tools they use, their stack. “Prefers Postgres and TypeScript” earns its place because it shapes every future reply.
  • Projects and context. What they are working on, the systems they own, the names that keep recurring.
  • Decisions and constraints. What they committed to, budgets, deadlines, hard limits. High consequence, low cost to keep.
  • Durable personal facts the user volunteered and would expect you to remember: their name, their role, their timezone.

The drop list is everything transient: the specific question they asked today, their fleeting mood, pleasantries, and anything that is true this minute but not next month. A good rule for the extractor: if it would still be useful to know this a month from now, keep it; if it is only about the current task, let conversation memory handle it and let it die with the session.

Updating and contradiction

People change, and this is the failure that separates memory that helps from memory that quietly sabotages. Someone tells the assistant in March they live in Munich. In September they mention they moved to Berlin. If your write path only ever appends, the store now holds both facts, and next time geography comes up the model gets “lives in Munich” and “moved to Berlin” side by side and has to guess. Sometimes it picks the stale one. There is no error. It is doing reasonable work with a store that contradicts itself.

The fix is that a new fact is not automatically a new row. Before you write, you compare it against what you already know for this user and decide what to actually do. A useful way to frame the decision is four operations: add it if it is genuinely new, update and supersede if it contradicts something, do nothing if you already know it, or delete if the user is retracting something.

// New fact in hand. Don't just append it. Compare against what you already
// know for this user, then pick one action.
function reconcile(newFact, existing) {
  const match = mostSimilar(newFact, existing);
  if (!match)                      return { op: "ADD" };   // genuinely new
  if (equivalent(newFact, match))  return { op: "NOOP" };  // already known
  if (contradicts(newFact, match)) return { op: "UPDATE", supersede: match.id };
  return { op: "ADD" };            // related, but not a replacement
}

The word to sit with is supersede, not delete. When “Berlin” replaces “Munich,” the clean move is to keep the old fact but mark it inactive with a timestamp, rather than erase it. Your live query already filters on superseded_at IS NULL, so the model only ever sees “Berlin,” but you keep the history. That history is worth having: it is an audit trail when a memory goes wrong, and it lets you answer “where did they used to live” if a task ever needs it. Blind-deleting throws that away; blind-appending poisons the present. Superseding keeps both the clean current view and the record of how you got there.

A new fact is not always a new row.BLIND APPENDfact: lives in Munichfact: moved to Berlinboth live, both retrieved, both sentmodel: so are you in Munichor Berlin? it has to guess.RECONCILE AND SUPERSEDElives in Munichsupersededlives in Berlinliveold kept for history, filtered from readsmodel: you are in Berlin.one live fact, no contradiction.Supersede, do not blind-delete: you keep a clean present and an audit trail of howyou got there. Blind-append keeps the contradiction and hands the guess to the model.
Blind append leaves the contradiction in the store and the model has to guess. Reconciling supersedes the old fact, so only the current one is live while the past is kept, inactive, for history.

Retrieving the right memories

Once the store has real volume, the game shifts from “do I have the fact” to “did I surface the right ones.” And here is the sharp edge that makes memory retrieval harder than it looks: injecting the wrong memory is worse than injecting none. An empty context makes the model ask a clarifying question. A wrong memory makes it confidently act on something false. Retrieve “loves spicy food” into a conversation about booking a mild-by-request dinner and you have actively made the answer worse than an assistant with no memory at all.

So relevance has to be more than raw similarity. A memory that is a little less similar but far more important, or far more recent, often deserves the slot more. A common shape is a blended score over three signals: semantic similarity to the current context, recency (older memories decay unless reinforced), and an importance weight that reflects how load-bearing the fact is.

// Relevance is not just similarity. A slightly less similar but more
// important or more recent memory often deserves the slot more.
const HALF_LIFE_DAYS = 60;

function score(memory, queryVec, now) {
  const sim = cosine(memory.embedding, queryVec);        // 0..1, semantic match
  const ageDays = (now - memory.updatedAt) / 86_400_000;
  const recency = Math.exp(-ageDays / HALF_LIFE_DAYS);   // fades over time
  return 0.6 * sim + 0.25 * recency + 0.15 * memory.importance;
}

Those weights are a starting point, not a law. Tune them against your own traffic. The important idea is that the ranking is deliberate, and you keep the injected set small and tight, the same discipline as RAG: a few sharp memories beat a big pile, because every irrelevant one you paste in spends context budget and risks steering the answer somewhere the user did not ask to go.

Pick a few relevant memories. A wrong one is worse than none.this turn: planning dinnerfor friends on Saturdayscore = 0.6 sim + 0.25 recency+ 0.15 importancecandidate memories, rankedvegetarian0.91 keepallergic to peanuts0.88 keeptop-k cutoff: only what is above goes into the promptuses Postgres0.12 dropbased in Berlin0.09 dropThe danger case: a stale memory scores just high enough to sneak above the line.Inject a two-year-old loves steak and the model plans a steakhouse for a vegetarian.Empty context asks a question. A wrong memory acts on a lie.Keep the injected set small and current. Precision here matters more than recall,because the cost of a wrong memory is a confidently wrong answer.
Each candidate memory is scored on similarity, recency and importance, not similarity alone. The top few relevant facts are injected. A stale or off-topic memory that sneaks above the cutoff derails the answer, which is why wrong is worse than none.

Forgetting on purpose

That recency term hints at something products get wrong by default: a memory store should forget. Not everything is worth keeping forever, and a store that only grows becomes both a cost problem and a correctness problem, because stale facts keep resurfacing long after they stopped being true.

Forgetting is mostly bookkeeping. Let low-value, unused memories decay: age them out, or drop the ones that never get retrieved and never get reinforced. Reinforce the ones that keep proving useful by bumping their recency or importance each time they are actually used, so the facts the user relies on stay sharp while the one-off trivia fades. This mirrors how human memory works, and it is a reasonable model to borrow.

The honest caveat, and it is an active research area as of 2026, is that decay handles the low-value memories well and struggles with the dangerous case: a high-importance fact that has quietly gone stale. “Allergic to peanuts” should never decay. “Working on the Q3 launch” should, once Q3 is over, but nothing in a similarity score tells you the launch shipped. There is no clean automated answer yet, which is the strongest argument for the thing the next section is really about: let the user see and correct their own memories, because they know what is stale and your decay heuristic does not.

Structured facts or freeform notes

Two shapes show up for what you store, and the right answer is usually both.

Privacy is not a footnote here, it is the design

Everything above is a data problem, and the moment you persist facts about a person across sessions, the weight of that problem jumps. A single stateless model call is transient. A memory store is a growing, durable profile of someone, sitting in your database, built quietly from things they said. That is a different category of responsibility, and it is heavier than anywhere else in this part of the course. The general handling rules from PII and safety all apply. Three of them apply harder.

Scope every memory to exactly one owner, and never cross the streams. Every memory carries a user_id. Every read filters on it, inside the query, before a single row is returned. This is the same authorization and multi-tenancy boundary you enforce on every other read, and the model gets no exemption for feeling like a magic text box. The nightmare here is specific and worse than a normal data leak: retrieve one user’s memories into another user’s prompt and you have not just exposed data, you have made the assistant confidently tell Person A intimate facts about Person B. A too-broad query, a cache keyed without the user id, a “similar users” feature that reaches across accounts: any of these leaks a person into someone else’s conversation.

// A deletion request has to reach every copy, not just the tidy one.
async function forgetUser(userId) {
  await db.query(`DELETE FROM memories WHERE user_id = $1`, [userId]);
  await db.query(`DELETE FROM memory_events WHERE user_id = $1`, [userId]);
  await cache.deleteByPrefix(`mem:${userId}:`);
  // and anything derived: vector index entries, exports, summaries
}

Consent, and a real off switch. Tell people the assistant remembers them, in plain language, not buried in a policy nobody reads. Let them turn it off. And do not silently feed their remembered facts into training or fine-tuning without separate, explicit consent, because a memory feature that quietly becomes a training pipeline is exactly the kind of thing that ends up in a headline.

View and delete, built in from day one. Users must be able to see what you remember about them and delete it, and increasingly the law requires it, not just good manners. The deletion has to actually reach every copy: the store, the event log, the cache, any exports. That only works if you designed for it, which is why deletion belongs in the schema on day one, not bolted on after a legal review. A memory the user can inspect and correct is also just a better feature, because they will fix the stale facts your decay heuristic could not.

One owner per memory. The filter is the whole safety story.Dana (user_id u_412)vegetarianbased in Berlinuses Postgresher partition, her reads onlySam (user_id u_907)allergic to shellfishbased in Torontoprefers Pythonhis partition, his reads onlyWHERE user_id = callerno cross-over, everyour memories (Dana’s own view)view and edit eachdelete everythingdeletion must reach the store, the log, the cache, and any exports.
A memory store partitioned strictly per user. Every read is filtered by the caller's id, so one user's memories can never reach another's prompt, and the owner gets a panel to view and delete what you hold.

See it survive a session

No model and no network below. The first panel is a canned conversation where the user states a couple of preferences. Run Extract facts and a toy extractor scans the text and writes structured memories into the store, exactly the write path from earlier. Now edit or delete a memory, then jump to next week’s session and ask a fresh question. The answer is assembled purely from whatever is in the store at that moment, so changing a memory changes the reply, and deleting the relevant one drops the assistant back into amnesia.

interactiveA memory that outlives the chat: extract, edit or delete, then ask next week

Nothing in that demo is smart. The extractor is a keyword scan and the “model” is a lookup table. That is the honest shape of the mechanism, and it makes the point cleanly: the assistant is only ever answering from what is in the store at the moment you ask. Continuity across sessions is not something the model does. It is something your store remembers and your read path pastes back in.

Summary

  • Conversation memory keeps one chat coherent by resending the transcript; long-term memory persists facts across sessions and brings the relevant ones back next time. It is the RAG loop pointed at facts about a person instead of documents.
  • The loop is extract, store, retrieve. A small model call pulls durable facts out of a conversation, you write them per user with provenance and timestamps, and on a later turn you retrieve the relevant few and inject them. Keep extraction off the hot path with a background job.
  • Extract selectively. Over-remembering bloats retrieval and hoards stale trivia; under-remembering is amnesia. Keep stable preferences, projects, decisions and constraints; drop transient chatter. Store a confidence with each fact.
  • Update, do not just append. When a new fact contradicts an old one, supersede the old one (mark it inactive, keep it for history) so the model sees one live fact, not a contradiction it has to guess through.
  • Retrieval relevance is the sharp edge: a wrong memory is worse than none, because it makes the model act confidently on something false. Rank by similarity, recency and importance, and keep the injected set small.
  • Let memories decay. Age out and prune unused, low-value facts; reinforce the ones that keep proving useful. Stale high-importance facts are the hard case, which is another reason to let users correct their own memory.
  • Privacy is the design, not a footnote. Scope every memory to one owner and filter every read on it so no user leaks into another’s prompt (authorization, multi-tenancy). Get consent, offer an off switch, and build view and delete in from day one so a deletion reaches every copy (PII and safety).
  • A poisoned memory is prompt injection that does not expire. Treat what is allowed to become a memory as a trust boundary, and keep memories inspectable and removable.