Caching Model Responses
A support bot I worked on cached its answers to save money. It worked. The bill dropped, the answers came back instantly, and everyone moved on. Three weeks later a customer asked how to cancel an order and got a crisp, confident walkthrough for cancelling a subscription instead. Different flow, different refund rules, wrong answer. It had been served that way, fast and cheap, to who-knows-how-many people, because the two questions were close enough in meaning that the cache decided they were the same question.
Nobody caught it for a week, because a wrong cached answer looks exactly like a right one. Same tone, same formatting, same speed. There was no error in the logs. There was just a fluent paragraph that happened to be about the wrong thing.
That is the whole tension of this topic. The cheapest, fastest model call is the one you never make, and caching cuts cost and latency harder than almost anything else you can do to an AI feature. But caching a model response is not caching a database row. A row is either the right row or a missing row. A cached answer can be subtly, confidently, invisibly wrong, and you served it on purpose.
You already know the general craft: TTLs, keys, invalidation, cache-aside, the stampede. That is all in caching layers and none of it is repeated here. This article is about the parts that are specific to model output, and the ways it bites differently.
Three different things people call “caching AI”
“We cache the AI” gets said about three genuinely different mechanisms. They live in different places, they key on different things, and they fail in different ways. Keeping them straight is most of the battle.
a hash of the whole input
the stored text, only on identical input
an embedding of the query
a neighbour’s answer if it clears a threshold
the token prefix, on their servers
the same output, billed cheaper and faster
The rest of the article takes them one at a time, then gets into the problems they share: invalidation, personalisation, and the fact that you are freezing one sample of a non-deterministic thing.
Exact-match caching: boring, and it works
Start with the one nobody argues about. If the exact same input comes in, hand back the exact same stored output. No model call at all.
This sounds too simple to matter, and for chatbots it mostly is, because two users rarely type byte-identical messages. But a huge amount of production AI is not a chatbot. It is a pipeline: classify this ticket, extract these fields, tag this image, moderate this comment. Those run at temperature: 0, the same handful of inputs show up constantly, and the same input genuinely should produce the same output every time. That is exact-match’s home turf, and there it is close to free money.
The entire mechanism is a hash and a key-value store. The only thing you have to get right is what goes into the key.
The rule: the key must contain everything that can move the output. Not just the user’s message. The model id, the temperature and other sampling params, max_tokens, the system prompt, and the tool definitions if you send any. Any one of those changing changes the answer, so any one of those changing has to change the key.
import { createHash } from "node:crypto";
function cacheKey(request) {
// Build the object in a fixed field order so the JSON is stable.
const canonical = JSON.stringify({
model: request.model,
temperature: request.temperature,
top_p: request.top_p,
max_tokens: request.max_tokens,
system: request.system,
tools: request.tools ?? null,
messages: request.messages,
});
return createHash("sha256").update(canonical).digest("hex");
}Then the wrapper is the same cache-aside shape you already use for a database, just with a model call as the expensive miss path:
async function cachedComplete(request) {
const key = `llm:${cacheKey(request)}`;
const hit = await cache.get(key);
if (hit) return JSON.parse(hit); // no model call at all
const response = await model.complete(request); // the paid call
await cache.set(key, JSON.stringify(response), { ttl: 86_400 });
return response;
}One quiet trap lives in that JSON.stringify. It serialises object keys in insertion order, so two logically-identical requests whose fields were built in a different order hash to different keys and never share a cache entry. Your hit rate silently drops and you never see an error. If any part of the input is an object whose key order you do not control, sort the keys before hashing. Canonicalise first, then hash.
Semantic caching: cache by meaning, and mind the knife-edge
Exact-match only fires on byte-identical input, which for real user questions almost never happens. “What is your refund policy,” “how do refunds work,” and “can I get my money back” are the same question wearing three outfits, and exact-match sees three different strings and calls the model three times.
Semantic caching fixes that by keying on meaning instead of bytes. You turn the query into an embedding, a vector that places similar meanings near each other in space, and you keep your past queries in a vector index. A new query gets embedded too, you find its nearest neighbour, and if that neighbour is close enough you return the answer you already have.
“Close enough” is measured with cosine similarity, which for two vectors is the dot product over the product of their lengths. It runs from -1 to 1, and for text embeddings a value near 1 means “basically the same thing.”
function cosine(a, b) {
let dot = 0, na = 0, nb = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i];
na += a[i] * a[i];
nb += b[i] * b[i];
}
return dot / (Math.sqrt(na) * Math.sqrt(nb));
}The lookup is one embedding call, one nearest-neighbour search, and one comparison against a threshold:
const THRESHOLD = 0.9; // tuned per route, never guessed
async function semanticLookup(query) {
const q = await embed(query); // a cheap embedding call
const { answer, score } = await vectorIndex.nearest(q); // top-1 neighbour
if (score >= THRESHOLD) {
return { hit: true, answer, score };
}
return { hit: false }; // fall through to the model
}Notice the embedding call on the hot path. A semantic cache is only a win when the embedding is far cheaper than the generation it saves, which it almost always is, but it is not free. You spend a small call and a vector search on every request, hit or miss, to skip an expensive one on the hits.
The threshold is the whole game
That THRESHOLD constant is where semantic caching lives or dies. Set it too high and almost nothing matches, so your hit rate collapses toward zero and you are paying for embeddings and a vector index that never save you anything. Set it too low and different questions start matching, and you serve one question’s answer to another. That is the war story from the top of this article, and here is exactly how it happens.
Be honest about the numbers, because the marketing around semantic caching is not. You will read about 90-plus-percent hit rates. In production, on real traffic, a threshold conservative enough to keep false hits under half a percent typically buys you a hit rate in the low tens of percent, not the nineties. A threshold around 0.9 is a common starting point as of 2026, but there is no universal right value. It depends on your embedding model and, more than anything, on how bad a wrong answer is for your particular feature.
So do not pick it by vibe. Build a small labelled set: pairs of queries that should match and pairs that look similar but should not, like the cancel-order and cancel-subscription pair. Sweep the threshold across that set, and pick the highest hit rate you can get while keeping false hits under the tolerance you actually have. This is a plain eval, and it is the difference between a semantic cache you can trust and one that is quietly lying to a slice of your users.
Play with exactly that trade-off. Type a question or tap an example, then drag the threshold. Watch the “cancel my order” question turn from a safe miss into a wrong-answer hit as you loosen it.
There is a security edge here too. Because the match is fuzzy, an attacker can sometimes craft a benign-looking question that lands near a poisoned entry, or probe your cache to see what other people have been asking. If untrusted users can write to the same semantic cache, treat it as an attack surface, the same wariness you bring to prompt injection. Do not let one user’s input silently become another user’s cached answer without a check.
Prompt caching: the provider’s cache, not yours
The third thing is different in kind, and the naming collision causes real confusion. Prompt caching is not you storing outputs. It is the provider caching the expensive internal computation for a repeated prefix of your input, and billing you less for the part it could reuse.
Here is the mechanism, because it explains everything about when it helps. Before a model generates anything, it has to read your entire prompt and compute a big pile of internal tensors from it (the key-value cache, one set per token). For a long prompt this “prefill” is often the slowest and most expensive part of the call. If your next request starts with the exact same tokens, the provider can reuse the tensors it already computed for that prefix instead of redoing the work. You still pay for the new tokens at the end. The shared beginning gets a steep discount.
The single design rule that follows: keep the static content at the front and byte-identical. Your system prompt, your tool definitions, a big reference document, the retrieved chunks for a long context. Put those first, unchanged, and put the one thing that varies, the user’s turn, at the end.
const response = await model.complete({
model: "some-model",
system: [
{ type: "text", text: BIG_STATIC_SYSTEM_PROMPT },
{ type: "text", text: LONG_REFERENCE_DOC, cache: true }, // mark the reusable tail
],
messages: [
{ role: "user", content: userQuestion }, // the only part that changes
],
});That cache: true marker is illustrative. As of 2026 the exact spelling differs by provider, and some providers cache eligible prefixes automatically with no marker at all once the prefix is long enough. The concept is stable even though the API is not.
The reason prompt caching is the safe one is worth stating plainly. It reuses the exact tensors the model would have computed anyway, so a cache hit produces the same distribution as a cache miss. It does not touch sampling. Temperature still applies, the model still draws a fresh token sequence, and you are not freezing anything. See sampling and temperature for why that draw is random in the first place. The cost you were spending on tokens for the repeated prefix just gets a lot cheaper.
The problems all three share
Underneath the differences, response caches inherit the two hard problems from ordinary caching, and both are sharper when the cached thing is a paragraph of fluent text.
Invalidation when the knowledge changes
A cached answer can go stale without a single character of it changing, because the world it describes changed. You cache “our refund window is 30 days.” Finance changes the policy to 14 days. The database is updated, the docs are updated, and your cache keeps confidently reciting 30 days to everyone who asks, in perfect prose, for as long as the entry lives. Nothing looks broken. It is just wrong now.
For answers that are grounded in a knowledge base or a set of documents, the clean fix is to fold a version of that source into the key, the same versioned-key trick from caching layers:
// Bump knowledgeVersion whenever the underlying policy or documents change.
const key = `llm:${knowledgeVersion}:${cacheKey(request)}`;Change the policy, bump the version, and every answer built on the old version is orphaned at once. No hunting down individual keys. If you cannot cleanly version the source, fall back to a TTL short enough that the worst-case staleness is something you can live with, and be honest in that estimate. A stale styling tip is fine for a day. A stale refund window is a support ticket, or a chargeback.
The personalisation trap, which is the dangerous one
This is the failure that turns a caching win into an incident. If an answer depended on one user’s private data, and you store it under a key that does not identify that user, the next person to ask the same question gets served the first person’s private answer.
In the security literature this is a cross-session leak, and it sits under sensitive-information disclosure in the current OWASP guidance for LLM apps. It is the same class of bug as the shared-dashboard cache from caching layers, except worse in one way: the leaked content is generated prose, so it can weave a user’s name, balance, order history, or health detail into a paragraph that reads like it was written for whoever receives it.
Two rules keep you safe. First, anything whose answer depended on private context must have the user or tenant id in the key, so one person’s answer can never be handed to another. That is authorization and multi-tenancy applied to your cache. Second, and more honest for genuinely personal answers, do not put them in a shared cache at all:
// If the answer used this user's private data, either scope the key to them...
const key = `llm:${userId}:${cacheKey(request)}`;
// ...or, for truly per-user answers with near-zero reuse, skip the cache entirely.
if (usedPrivateContext) return await model.complete(request);A per-user answer that each user asks once has a hit rate near zero anyway, so caching it buys you almost no speed while creating a real chance of a leak. That is a bad trade in both directions.
You are freezing one sample
There is a subtler thing worth naming. A model at any temperature above zero is a sampler. Ask it the same question twice and you get two different phrasings, sometimes two genuinely different answers. The moment you cache a response, you take one of those draws and pin it as the answer for everyone who matches, possibly for a long time.
Usually that is fine, and often it is what you want. But be aware of what you chose. If the value of the feature was variety, a brainstorm, a greeting, a creative line, then a cache serves the same “spontaneous” output to thousands of people and the spontaneity is gone. And if the one sample you happened to freeze was the occasional weird or subtly-wrong one, you did not cache a good answer, you cached that specific mistake and gave it a long life. For a temperature: 0 classifier this is a non-issue. For anything generative, know that a cache is a snapshot of one roll of the dice.
When not to cache
Caching model output is a hammer, and a lot of things are not nails. Skip it, or cache with real caution, when any of these are true.
- The answer is personal. It used one user’s private context. Either scope it hard to that user or do not cache it. A cross-user leak is not a performance regression, it is a breach.
- The answer is time-sensitive. Anything that depends on “now”: prices, availability, a live status, “what happened today.” A cache is a promise that a slightly old answer is good enough, and for these it is not.
- A subtly-wrong hit is dangerous. Medical, legal, financial, safety, or security-relevant answers. The cost of a confident wrong answer is high enough that shaving latency is not worth even a small false-hit rate. If you cache here at all, use exact-match, never semantic.
- Reuse is near zero. If each input shows up roughly once, you are paying for storage and lookups to serve almost no hits. The cache is pure overhead.
The number that tells you whether any of this is working is not just the hit rate. It is the hit rate measured against the cost of a wrong hit. A 60% hit rate on a refund bot sounds great until you learn that 2% of those hits are the wrong policy. Watch the hit rate, and separately sample real hits for correctness, the same instinct as an eval pointed at your cache instead of your model. A cache that is fast and occasionally lying is often worse than no cache at all, because it fails silently and it fails in prose.
Summary
- Three different mechanisms hide behind “we cache the AI.” Exact-match and semantic caches store your outputs; prompt caching is the provider reusing the computation for a repeated prefix. They key on different things and fail in different ways.
- Exact-match is boring and effective for repeated identical calls. The whole job is the key: hash the model, all sampling params, the system prompt, the tools, and the messages. Miss one and a changed setting serves a stale answer.
- Semantic caching matches by meaning via an embedding and a nearest-neighbour search. The similarity threshold is the whole game: too high and hit rate collapses, too low and you serve one question’s answer to another. Tune it on a labelled set, not by feel, and re-tune whenever the embedding model changes.
- Prompt caching is provider-side, safe, and cheap: keep the static content at the front and byte-identical, put the varying user turn last. It reuses computation without changing the output, so it never freezes a sample. Exact numbers move fast; check current docs.
- The shared hard problems bite harder in prose: invalidation (version the key by your knowledge source so a policy change orphans old answers) and personalisation (scope per-user answers to the user or do not cache them, or you leak one person’s private data to the next).
- A cached response is a frozen sample of a non-deterministic model. Fine for a
temperature: 0classifier, worth thinking about for anything generative. - Do not cache the personal, the time-sensitive, or the dangerous-when-wrong. Measure hit rate and the correctness of your hits. A fast cache that lies is worse than a slow one that does not.