Query Rewriting and Expansion
The support bot answered the first question perfectly. “How long is the refund window on the Basic plan?” got a clean “30 days,” pulled straight from the right doc with a citation. Then the customer typed the obvious next thing: “and for the higher tier?” The bot retrieved three chunks about API rate limits and confidently made up a refund policy that did not exist.
Nothing was wrong with the model. Nothing was wrong with the index. The bug was that we embedded the string “and for the higher tier?” and asked a vector store to find documents near it. That string has no verb, no topic, and does not contain the word “refund” or “Pro” anywhere. It is meaningless on its own. The retriever did exactly what we asked and returned garbage, because we handed it garbage.
You built the whole retrieval pipeline in the RAG pipeline: chunk, embed, nearest-neighbour search, stuff the results into the prompt. Every piece of that runs on one input, the text you drop into the “embed the question” box. The entire system is only ever as good as that string. And the string you get for free, the user’s literal words, is frequently the worst possible search query. Fixing it before retrieval runs is one of the cheapest, highest-return things you can do to a RAG system. It is also mostly unglamorous string manipulation dressed up with one extra model call.
Why the raw query underperforms
Three failure shapes come up again and again. Once you can name them, you know which technique to reach for.
The context-dependent follow-up. “Does it work on the new one?” “What about pricing?” “and for the higher tier?” In a conversation these are perfectly clear to a human, because the topic lives in the previous turns. To a retriever that sees only this one string, they are noise. A model has no memory between calls; the retriever has even less. Whatever the pronoun points at, whatever “it” and “that” and “the higher tier” mean, is sitting in the conversation history and never made it into the search.
The vocabulary gap. A user types “how do I turn it off.” Your documentation says “disabling auto-renewal.” Embeddings close some of that gap, since vector search matches meaning rather than exact words, but not all of it. A four-word question and a four-hundred-word policy page live in genuinely different regions of the vector space. The question is short and shaped like a question. The document is long and shaped like a statement of fact. They are about the same thing and still do not sit as close together as you would hope.
The multi-part question. “How do I cancel and can I get a refund?” is two searches wearing a trench coat. Cancellation lives in one chunk, the refund policy in another. Embed the whole sentence and you get a single vector that averages both intents into a blurry midpoint that matches neither chunk well. You retrieve something vaguely on-topic for both and nail neither.
The rest of this article is a toolbox, one tool per failure shape, plus the discipline to not use all of them all the time.
Rewriting a follow-up into a standalone query
This is the one you should reach for first, because in any chat product it fixes the most common and most embarrassing failure. The move is simple: before you retrieve, make a quick model call that rewrites the latest turn into a self-contained query, using the conversation so far to fill in whatever the user left implicit.
In production chat traffic, well over half of follow-up messages carry an unresolved reference, a “that,” an “it,” an elided topic. Retrieval on the raw turn falls apart the moment the conversation gets past turn one. The rewrite resolves the pronouns and pulls the topic forward.
In code it is a small function. Feed it the recent history and the latest turn, ask a fast model for the standalone query, and hand that to your existing retriever.
// A cheap, fast model turns a context-dependent turn into a standalone query.
async function rewriteForRetrieval(history, latestTurn) {
const prompt = [
"Rewrite the user's last message as a standalone search query.",
"Resolve pronouns and implicit references using the conversation.",
"Keep it short. Output only the query, nothing else.",
"",
formatRecentTurns(history), // the last few turns, not the whole transcript
"User: " + latestTurn,
].join("\n");
const out = await small.generate({ prompt, temperature: 0 });
return out.text.trim();
}Two details earn their keep. Run the rewrite at temperature 0, because you want the same query every time, not a creative reinterpretation. And feed it recent turns, not the entire chat, both to keep the call cheap and because the topic almost always lives in the last exchange or two. This is exactly the history you are already managing in conversation memory; the rewrite is one of its main customers.
Then the pipeline barely changes. You slot the rewrite in front of retrieval and everything downstream is identical:
// Only rewrite when it will help. A first turn has no context to resolve.
const query = isFollowUp(turn) ? await rewriteForRetrieval(history, turn) : turn;
const chunks = await retrieve(query); // embed + nearest-neighbour, unchangedQuery expansion: widen the vocabulary
Expansion adds terms. You take “cancel” and also search for “terminate,” “unsubscribe,” “end subscription.” The goal is to catch documents that mean the same thing in words the user did not happen to type.
Here is the honest scoping. Pure vector search already handles synonyms fairly well, because “cancel” and “terminate” sit near each other in embedding space by construction. So expansion buys you the most on the keyword side of retrieval, the exact-token, BM25 leg that matches strings literally and does not know “cancel” and “terminate” are cousins. If you run hybrid search, expanding the query is mostly a gift to the lexical half.
// Expansion helps the keyword leg most: BM25 matches tokens literally.
const expanded = await small.generate({
prompt: "List 4 alternative phrasings and synonyms for this search, comma-separated: " + query,
temperature: 0,
});
const keywordQuery = query + " " + expanded.text; // union of terms for BM25Do not overdo it. Pile on twenty loosely related terms and you dilute the query until it matches everything weakly, which is its own kind of miss. A handful of tight synonyms, added to the keyword leg only, is the whole technique.
Multi-query: search from several angles
One phrasing sees one slice of the corpus. A different phrasing of the same question sees a different slice. Multi-query leans into that: ask the model for a few distinct rewrites of the question, retrieve for each one in parallel, then merge the result lists into a single ranking.
The merge is not a new idea. It is reciprocal rank fusion, the same rank-based combine you use to fuse keyword and vector lists. Here you are fusing several vector lists instead, but the logic is identical: a document that ranks well across multiple lists beats one that topped a single list and sank in the others.
// Ask for a few phrasings, retrieve each in parallel, fuse the lists by rank.
const phrasings = await expandToQueries(question, { n: 3 });
const lists = await Promise.all(phrasings.map(retrieve));
const merged = reciprocalRankFusion(lists); // same RRF as hybrid searchThis combination, multi-query plus rank fusion, is what some people call RAG fusion. It is worth the extra retrievals when your problem is recall: the right chunk exists in your index, but a single phrasing keeps failing to surface it. It does nothing for a corpus that simply does not contain the answer. Multi-query cannot retrieve what was never ingested.
HyDE: search with a hypothetical answer
This one is a genuinely clever trick, and it is worth slowing down for.
Go back to the vocabulary gap. Questions and answers live in different neighbourhoods of embedding space. A question is short, interrogative, and full of question words. Your documents are long, declarative statements of fact. So when you embed a question and compare it to documents, you are comparing two things that are shaped differently, and the geometry works against you even when they are about the exact same topic.
HyDE, hypothetical document embeddings, sidesteps that. Instead of embedding the question, you ask the model to draft a fake answer to it, a short passage that looks like the kind of document that would answer the question. Then you embed that hypothetical passage and search with its vector. Because the draft is shaped like a document, it lands much closer to your real documents than the question ever would.
The part that trips people up: the hypothetical answer does not need to be correct. It will often be wrong, because the model is guessing from its frozen training and knows nothing about your specific refund window or your internal product names. That is fine. You are not showing this draft to anyone. You are using it as a search probe, and even a factually wrong passage tends to use the right vocabulary and the right sentence shapes, which is all the retriever needs to find the genuine documents nearby.
// Draft a fake answer, embed THAT, search with it. The draft is a search
// probe, not an answer. Its accuracy does not matter; its shape does.
async function hydeRetrieve(question) {
const draft = await small.generate({
prompt: "Write a short, factual-sounding passage that answers: " + question,
temperature: 0,
});
const vec = await embed(draft.text); // embed the hypothetical, not the question
return nearestNeighbours(vec, { k: 8 });
}HyDE shines brightest when your retriever is generic and untuned and the domain is one the model reasonably knows. The more you have invested in a domain-specific embedding model, the less HyDE tends to add, because you have already closed the question-answer gap it exists to bridge.
Decomposition: break the question apart
Some questions are not one search, they are several. “Does the Pro plan include SSO, and can I export my data as CSV?” contains two independent facts that live in two different documents. No single retrieval and no single rewrite will serve both well. The fix is to split the question into sub-questions, retrieve for each one separately, and compose the pieces into one answer.
const subQuestions = await decompose(question);
// -> ["Does the Pro plan support SSO?", "Can I export data as CSV?"]
const evidence = await Promise.all(subQuestions.map(retrieve));
const answer = await generate({ question, context: evidence.flat() });There are two flavours, and they cost differently. Parallel decomposition, like the example above, splits into independent sub-questions you can retrieve for all at once. Sequential decomposition is for genuinely multi-hop questions where you cannot even write the second search until you have answered the first. “What is the refund window on whichever plan includes SSO?” needs you to find the SSO plan, read that it is Pro, and only then search for the Pro refund window. Sequential is strictly more expensive, because the retrievals cannot run in parallel and each hop is a round trip. Reserve it for questions that truly chain.
The bill: do not rewrite everything
Every technique on this page adds at least one model call before retrieval even starts. That call sits on the critical path, so the user waits through a full round trip they did not used to wait for, and you pay for the tokens. Stack them naively, always multi-query, then HyDE, then decompose, then rerank, and you have bolted three or four extra model calls and a few hundred milliseconds onto every single question, most of which needed none of it.
So treat these as targeted fixes, not defaults. A few rules that hold up:
- Use a small, fast model for rewrites. The rewrite is not the hard part of your system. A cheap model at temperature 0 does it fine, and it keeps the added latency and cost small.
- Gate the expensive techniques. Condensing a follow-up is cheap and almost always pays off in a chat, so do it whenever the turn is a follow-up. Multi-query, HyDE, and decomposition are heavier, so trigger them only when a cheap signal says the question needs them: it is multi-part, or it is long and compositional, or the first plain retrieval came back weak.
- Retrieve first, then decide. A pattern worth stealing: run the plain, one-shot retrieval first. If the top result’s score is strong, you are done, no rewrite, no extra call. Only when the top score comes back weak do you escalate to the expensive path. Most questions never need it.
// Adaptive: pay for the rewrite only when the cheap path is weak.
let chunks = await retrieve(question);
if (topScore(chunks) < THRESHOLD) {
const phrasings = await expandToQueries(question, { n: 3 });
chunks = reciprocalRankFusion(await Promise.all(phrasings.map(retrieve)));
}Prove it actually helped
Here is the trap. You add HyDE, you try a few questions by hand, the answers feel sharper, you ship it. Three weeks later you find out it quietly made retrieval worse on the long tail and you never noticed, because “feels sharper” is not a measurement.
Query rewriting is a change to retrieval, so you measure it the way you measure any retrieval change, the way chunking taught: take a set of real questions, note which chunk should come back for each, and check whether it lands in the top few results. That fraction is your recall. Run it on the raw query and again on the rewritten query, and compare the two numbers. If recall goes up, keep the rewrite. If it does not move, you just added latency and cost for nothing. If it goes down, and it sometimes does, a rewrite that drops the one word that actually mattered, you back it out.
// Did rewriting move retrieval recall, or did it just cost money?
function recallAtK(cases, searchQuery, k) {
let hits = 0;
for (const c of cases) {
const ids = search(searchQuery(c)).slice(0, k).map((r) => r.id);
if (ids.includes(c.goldChunkId)) hits++;
}
return hits / cases.length;
}
const raw = recallAtK(cases, (c) => c.question, 5);
const rewritten = recallAtK(cases, (c) => rewrite(c.question, c.history), 5);
// ship the rewrite only if rewritten > raw by a margin you care aboutThis is the same discipline as the rest of evals: decide what “good” means, put a number on it, and let the number, not the vibe, decide what ships. Half the fashionable rewrite techniques do nothing for your particular corpus. The only way to know which half is to measure on your data.
See it work
Below is a tiny, fully simulated retriever. There is no model and no network here; the “rewrite” is canned logic and the “search” is plain word-overlap against five fixed documents. The point is the mechanism, not real inference. The conversation ends on a vague follow-up. Search on the raw follow-up and watch it whiff. Hit rewrite, then search again on the standalone query, and watch the right document jump to the top.
Word-overlap is a crude stand-in for real embeddings, but the shape of the lesson is exactly right. The raw follow-up shares no words with any document and retrieves nothing useful. The rewritten query, carrying the topic that was hiding in the history, matches the correct chunk immediately.
Summary
- The retriever only ever sees the string you hand it. The user’s literal words are often a poor search query, and improving that string before retrieval is one of the cheapest wins in RAG.
- Rewriting turns a context-dependent follow-up into a standalone query using the conversation history. It is the highest-value fix for any chat product. Rewrite at temperature 0, search with the rewrite, but still answer from the user’s real turn.
- Expansion adds synonyms and related terms. It mostly helps the keyword leg of hybrid search, since vector search already handles synonyms. A few tight terms, not twenty loose ones.
- Multi-query generates several phrasings, retrieves each, and merges with reciprocal rank fusion. Reach for it when recall is the problem and the right chunk keeps slipping past a single phrasing.
- HyDE embeds a hypothetical answer instead of the question, because an answer-shaped probe lands nearer your documents. The draft can be factually wrong and still work, but on niche, fact-bound domains it can steer retrieval off a cliff. Measure it.
- Decomposition splits a compound question into sub-questions, retrieves for each, and composes. Parallel is cheap; sequential multi-hop is expensive, so reserve it for questions that truly chain.
- Every rewrite is an extra model call on the critical path. Use a small model, gate the heavy techniques behind a cheap signal, and consider retrieving first and only escalating when the top score is weak.
- Prove it helped. Measure retrieval recall on raw versus rewritten queries with evals. Plenty of fashionable rewrite tricks do nothing, or worse, for your particular corpus.