The RAG Pipeline End to End
The finance team cut our refund window from 30 days to 14. Someone updated the help doc the same afternoon, so by 3pm the source of truth was correct. The support bot kept telling customers “30 days” for another nine days.
Nothing was broken. The model was fine, the prompt was fine, the doc was right. The bot was faithfully answering from a copy of the old paragraph that no longer existed anywhere a human could see it: a vector we had embedded back when the doc still said 30. The job that turns docs into vectors ran on a weekly cron, and it had not run yet.
That bug is this whole article in miniature. A retrieval system has two separate clocks, and if you do not understand which one is which, it will hand a model perfect-looking context that is quietly wrong and you will spend your afternoon debugging the model instead of the pipeline. Retrieval-augmented generation, RAG, is the thing that lets a model answer from your data instead of its frozen training. It is also the thing that breaks at five different joints, and the failures all look the same from the outside: a confident, fluent, wrong answer.
Retrieve, don’t retrain
A language model’s weights are frozen the day training ends. It never saw your internal docs, your ticket history, last night’s incident writeup, or the refund policy you changed at 3pm. Ask it about any of that and it will either admit it does not know or, more often, invent something plausible.
You have two ways to close that gap. You can bake the knowledge into the weights by fine-tuning, which is slow, expensive, has to be redone every time the data changes, and still leaves you with a frozen snapshot. Or you can put the right text in the prompt at the moment you ask the question, so the model reads it and answers from it. That second option is RAG, and for facts it wins almost every time.
Here is the part that should feel familiar: RAG is not a new algorithm. It is the pieces you have already met in this chapter, wired together. Chunking to cut documents into retrievable passages. Embeddings to turn each passage into a vector. Cosine similarity and nearest-neighbour search to find the closest ones. A vector store to hold them. Hybrid search and reranking to get the ranking right. RAG is what you call it when you assemble those into a flow that ends at a model. The model’s job shrinks from “know everything” to “read these five passages and answer the question.”
Two phases, two clocks
The single most important thing to hold in your head is that RAG runs on two clocks, and they meet at exactly one place.
The offline phase is a data pipeline. It runs when your data changes, not when a user shows up. It loads a source document, chunks it, embeds each chunk, and writes the vectors plus metadata into the store. It can be slow. It can run as a background job on a queue, take thirty seconds a document, and nobody waits on it.
The online phase is a request handler. It runs on every single question, with a user staring at a spinner. It embeds the question, retrieves the nearest chunks, stuffs them into the prompt as context, asks the model, and returns the answer with citations. It has to be fast.
They share one thing: the vector store. Ingestion writes to it, queries read from it. That shared store is why my refund bot broke. The online phase was reading correctly. The offline phase had not written the new paragraph yet.
The offline half is a few lines once the helpers exist. Note the upsert: re-ingesting a changed document overwrites its old vectors instead of leaving stale ones behind, which is the fix for the bug I opened with.
// Offline. Runs when a document changes, not per request.
// Turn source text into rows the query phase can search.
async function ingest(doc) {
const chunks = chunkDocument(doc.text); // structure-aware, see /chunking
const vectors = await embed(chunks); // batched, see /embeddings
for (let i = 0; i < chunks.length; i++) {
await db.query(
`INSERT INTO chunks (doc_id, ord, text, embedding, source, updated_at, tenant_id)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (doc_id, ord) DO UPDATE
SET text = EXCLUDED.text,
embedding = EXCLUDED.embedding,
updated_at = EXCLUDED.updated_at`,
[doc.id, i, chunks[i], vectors[i], doc.source, doc.updatedAt, doc.tenantId]
);
}
}The online half embeds the question, pulls the nearest chunks, and asks the model. The retrieval itself is one parameterised query, scoped to the caller so a tenant can only ever see their own rows.
-- Online. The nearest chunks to the query embedding, scoped to the caller.
SELECT id, text, source, ord
FROM chunks
WHERE tenant_id = $1
ORDER BY embedding <=> $2 -- cosine distance; smaller is nearer
LIMIT $3;One question, start to finish
Watch a single question move through the online phase, because the “augment” step is the one people wave their hands at and it is the whole trick.
Someone asks “How do I cancel, and can I get a refund?” You embed that sentence with the same model you embedded the documents with (use a different one and the geometry is meaningless). You retrieve the top few chunks by distance. Then you build a prompt that literally pastes those chunks in as context, numbered, and tells the model to answer only from them and cite the numbers. The model reads real, current text and writes an answer that points back at its sources.
In code the augment step is a map and a join. There is no magic in it, which is the point. You are building a string.
async function answer(question, ctx) {
const [qVector] = await embed([question]);
const chunks = await retrieve(qVector, { tenantId: ctx.tenantId, k: 5 });
// Augment: number the chunks and paste them in as context.
const context = chunks
.map((c, i) => `[${i + 1}] (${c.source}) ${c.text}`)
.join("\n\n");
const messages = [
{ role: "system", content:
"Answer using only the context below. Cite sources as [n]. " +
"If the context does not contain the answer, say you don't know." },
{ role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` },
];
const reply = await generate(messages); // the model call, see /calling-a-model
return { reply, cited: chunks }; // keep the chunks for citations and debugging
}Two things earn their keep here. The role structure comes straight from messages and roles: the instruction to stay grounded goes in the system message, the retrieved context and question go in the user message. And you return the retrieved chunks alongside the reply, always. You need them to render citations, and you need them even more when the answer is wrong and you have to figure out why.
See the loop run
Below is the whole online phase, fully simulated in the page. A tiny corpus is already chunked and embedded with hand-assigned toy vectors. Pick a question and a fake embedder turns it into a vector, cosine finds the nearest chunks, you watch them get pasted into the prompt, and a grounded answer comes back with citations pointing at exactly the chunks it used. No model and no network are involved. Try the last question, the one about phone support, and watch retrieval come up empty and the system refuse instead of inventing.
That demo is the happy path, and it is genuinely most of a working RAG system. The rest of this article is about the ways it stops being the happy path.
It breaks at every joint
RAG has more moving parts than a plain model call, and every joint between them is a place the whole thing can silently fail. The maddening part is that the output looks identical no matter which joint broke: a fluent, confident answer that happens to be wrong. You cannot tell from the answer alone whether the chunker mangled the text, the retriever missed, the window overflowed, the vectors were stale, or the model ignored good context. So it is worth knowing all five by name.
Go through them, because the fix is different for each.
Bad chunking retrieves fragments. If the offline phase cut a question away from its answer, or stripped a table of its header, then even flawless retrieval hands the model a fragment that cannot answer anything. This is the highest-impact failure and it is upstream of everything, which is why chunking gets its own article. A retrieved chunk is read alone, so it has to make sense alone.
A weak retriever misses, and the ceiling is set right there. If the chunk that holds the answer is not in the top-k you fetched, the model never had a chance. It cannot reason its way to text it was not given. This is why pure vector search whiffing on an exact error code matters so much, and why hybrid search and a reranker exist: to get more of the right chunks into the shortlist. Retrieval recall is a hard ceiling on the whole system. Nothing downstream can lift it.
Too many chunks bury the answer. The instinct when answers are weak is to retrieve more chunks, on the theory that more context is safer. It is often the opposite. Every chunk you add spends context budget and, worse, dilutes attention. Models attend well to the start and end of a long context and worst to the middle, a U-shaped curve that a 2024 study named “lost in the middle,” and follow-up work through 2026 has shown accuracy also drifts down simply as the input grows longer, evidence favourably placed or not. Bury the one good chunk among fifteen mediocre ones and the model can miss it even though it was technically present. Fewer, sharper chunks beat a big pile almost every time.
Stale vectors serve yesterday’s truth. Embeddings are derived data. The moment the source changes, the vector is a lie until you re-embed. My refund bot lived here. The fix is to treat ingestion as something that reacts to change, not a weekly batch: re-embed on edit, or run a worker that fills in rows where the content hash moved. And when you switch embedding models you re-embed everything, because vectors from two different models are not comparable.
Even with perfect context, the model can ignore it. This is the humbling one. You can retrieve exactly the right chunk, place it perfectly, and the model still answers from its own training and contradicts the text in front of it. Research calls these context-faithfulness failures, and a common flavour is post-rationalisation: the model decides the answer from memory, then scans the context for tokens that seem to back it up. This is what grounding and citations fights, by forcing the model to point at the specific passage for each claim so an unsupported one becomes visible. It reduces the problem. It does not delete it.
The model is usually not the problem
Here is the working habit that separates people who ship RAG from people who thrash on it. When an answer is wrong, your instinct will be to edit the prompt or swap in a bigger model. Resist it. Look at what retrieval actually fed the model first.
Nine times out of ten, the model answered perfectly reasonably given what it was handed, and what it was handed was junk: a severed sentence, a header-less table, the wrong passage, or nothing relevant at all. “The model gave a wrong answer” almost always decodes to “retrieval failed.” You debug retrieval by inspecting the retrieved set, and it takes about five lines.
// When an answer is wrong, print what retrieval fed the model
// BEFORE you touch the prompt or reach for a bigger model.
const { reply, cited } = await answer(question, ctx);
console.table(cited.map((c, i) => ({
rank: i + 1,
source: c.source,
score: c.score?.toFixed(3),
preview: c.text.slice(0, 60),
})));
// Does the passage you KNOW holds the answer show up at all?
const goldPresent = cited.some((c) => c.id === expectedChunkId);
if (!goldPresent) console.warn("retrieval miss: the answer was never in the context");Run that and one of two things is true. Either the right chunk is in the list, in which case you have a genuine generation problem and now it is worth looking at the prompt. Or, far more often, it is not in the list at all, and no amount of prompt engineering was ever going to help, because you were asking the model to answer from text it never received.
Wire this into your tracing so every answer in production carries the retrieved chunk ids and scores as attributes. Then when a bad answer comes in, you are reading a trace instead of trying to reproduce the exact query, and the “was the right chunk even there” question is answered in seconds.
Isn’t a huge context window enough now?
Fair question, and it comes up every time windows get bigger. If a model can read a million tokens, why not skip retrieval and paste the entire knowledge base into every prompt? For a while in 2026 this was a genuinely live debate, usually under the headline “RAG is dead.”
RAG is not dead. What died is the lazy 2023 version where you chunk blindly, dump it in a vector database, and hope. The architecture is more common than ever, and there are concrete reasons it does not go away when windows grow.
There is also the quieter point from the failure map: a bigger window is not automatically a better answer. More context spends more money and invites the lost-in-the-middle and context-rot failures. Sending less, but more relevant, context usually beats sending everything, on both accuracy and cost. That is the whole argument for retrieval in one line.
On build versus buy: you can run the entire pipeline on Postgres with pgvector sitting next to your application tables, which is the right first move for most teams up to a few million vectors and keeps everything in one database you already operate. Reach for a dedicated vector store or a managed retrieval platform when you need very large scale, strict tail latency, or native hybrid search you do not want to assemble yourself. Either way the concepts are identical. Do not let a vendor convince you RAG is a product you buy. It is a pipeline you understand.
Measure the two phases apart
The reason a wrong answer is so confusing is that it could have failed at retrieval or at generation, and they look the same. So you measure them separately. This is not optional polish. A RAG system you cannot measure is one you tune by vibes, and vibes do not survive a re-index.
For retrieval, the metrics are about the chunks. Context recall: of the chunks that were needed to answer, how many did you actually fetch? Context precision: of the chunks you fetched, how many were relevant? Recall is the one that catches the “the answer was never retrieved” failure, and it is the single most valuable eval in a RAG system. You can write it in an afternoon: a list of questions, each tagged with the chunk that should come back, and a check for whether it landed in the top-k.
// Retrieval eval: for each question, was the right chunk retrieved at all?
// This is the number that chunking, embedding and hybrid search all move.
async function retrievalRecall(testset, k = 5) {
let hits = 0;
for (const { question, goldId } of testset) {
const got = await retrieve(await embedOne(question), { k });
if (got.some((c) => c.id === goldId)) hits++;
}
return hits / testset.length; // fraction where the answer was even reachable
}For generation, the metrics are about the answer given the context. Faithfulness: is every claim in the answer supported by the retrieved chunks, or did the model invent something? Answer relevance: does the answer actually address the question? These are usually scored by a separate model acting as a judge, and as of 2026 a faithfulness score above roughly 0.85 on a zero-to-one scale is decent, while anything under 0.7 means the model is hallucinating past its context often enough to worry about.
The reason to keep them apart is diagnostic. Imagine faithfulness looks great but users complain answers are incomplete. If you only measured end to end you would be stuck. Measured separately, you see recall is low: the retriever was missing the second relevant chunk on multi-part questions, and the generator was faithfully answering from the partial context it did get. High faithfulness, low recall, and the fix is entirely in the retrieval half. This is the retrieval-augmented shape of everything in evals: decide what “good” means, measure it, and let the number tell you which half to fix.
Summary
- RAG makes a model answer from your data instead of its frozen training, by putting the right text in the prompt at query time. It is not a new algorithm, it is this chapter’s pieces (chunking, embeddings, similarity, vector search, hybrid and rerank) assembled into a flow.
- It runs on two clocks that meet at the vector store. Offline ingestion (load, chunk, embed, store) runs when data changes and can be slow. Online query (embed, retrieve, augment, generate, cite) runs per request and must be fast.
- The augment step is plain string assembly: number the retrieved chunks, paste them into the prompt as context, and instruct the model to answer only from them and cite the numbers.
- It breaks at five joints, all with the same symptom, a confident wrong answer: bad chunking retrieves fragments, weak retrieval misses the chunk entirely, an overstuffed window buries the answer in the middle, stale vectors serve outdated text, and the model can ignore even perfect context.
- There is a sixth, security joint: a poisoned knowledge base is indirect prompt injection, and over-broad retrieval leaks across tenants. Filter permissions inside the query and treat retrieved chunks as data, never instructions.
- “The model gave a wrong answer” almost always means “retrieval failed.” Debug retrieval first: print the fetched chunk ids and scores and check whether the passage that holds the answer was even in the set. Trace it in production.
- Long context did not kill RAG. Retrieval wins on cost, freshness, scale, citations, and access control, and a bigger window invites lost-in-the-middle failures. Send less but more relevant context.
- Evaluate the two phases separately. Retrieval: context precision and recall. Generation: faithfulness and answer relevance. A wrong answer is one or the other, and measuring end to end hides which.