Grounding, Citations and Hallucination
A support bot we shipped once told a customer, in three calm and well-organised paragraphs, exactly how to expense their subscription: open Billing, click the Reimbursements tab, upload the receipt. That tab has never existed. The customer went looking for it anyway, could not find it, and filed the exact ticket the bot was supposed to prevent.
The bot was not broken. It was not even confused. It had done its one job perfectly, which is to produce the most plausible continuation of the conversation. A tidy help article for a reimbursements page is an extremely plausible thing for a billing bot to write. That the page does not exist was never something the model was in a position to check.
That is the whole problem in one story. A confident, fluent, completely wrong answer is indistinguishable, by tone alone, from a correct one. And a wrong answer delivered with total confidence is the thing that gets an AI feature pulled from production, because it is worse than a hundred honest “I don’t know”s. Grounding is your main defence against it, and it is the reason the RAG pipeline exists at all.
Why a model makes things up
Go back to what a language model actually does when you call it. It reads the tokens so far, predicts a probability distribution over the next token, samples one, and repeats. That is the entire mechanism. There is no step in there where it consults a database, checks a fact against reality, or asks itself whether it genuinely knows the answer.
So look at what “hallucination” means at that level. Ask the model for the method to cancel a Nimbus subscription and it generates the most likely continuation. nimbus.cancelSubscription() is a gorgeous continuation. It matches every pattern the model has ever seen: an object, a dot, a verb-then-noun method name, parentheses. Whether that method exists in your SDK is a completely separate question, and the sampling loop never asks it. A plausible falsehood and a plausible truth look identical from the inside. Both are just high-probability sequences of tokens.
There is a second reason, and it is about how models are trained and scored rather than how they run. When you grade a model on a benchmark, “I don’t know” is marked wrong. A confident guess that happens to land is marked correct. Multiply that over millions of examples and a model that guesses boldly outscores a model that admits doubt, even though the bold one is wrong more often. We have, fairly literally, trained models to bluff. There is active work on rewarding calibrated uncertainty instead, giving credit for a well-placed “I’m not sure”, and it helps, but the default behaviour you get from a model out of the box still leans toward the confident guess.
Grounding: pin the answer to sources
Grounding is the move that turns “predict something plausible” into “answer from this specific text, and only this text.” You retrieve the relevant passages, paste them into the prompt as context, and instruct the model to answer strictly from what you gave it, and to say it does not know when the context does not cover the question.
That last clause is the one people skip, and it is the important one. A model told only “answer from the context” will still stretch to fill silence. A model explicitly given permission to fail will take it. Refusal is not the system breaking. It is the system working.
In code, grounding lives almost entirely in the system message. The retrieval half (embed, search, rank) is the RAG pipeline; here is the instruction that turns retrieved text into a grounded answer.
const system = [
"Answer the question using ONLY the numbered context below.",
"After each sentence, cite the context it came from as [n].",
"If the context does not contain the answer, reply exactly:",
'"I could not find that in the provided sources." Do not guess.',
].join("\n");
const context = chunks
.map((c, i) => `[${i + 1}] (${c.source}) ${c.text}`)
.join("\n\n");
const messages = [
{ role: "system", content: system },
{ role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` },
];Does it work? Measurements vary a lot by task, but the shape is consistent across recent studies: retrieval plus a grounding instruction removes most hallucinations, on the order of 75 to 90 percent in reported evaluations. A bare instruction with no retrieved context (“please cite your sources”, “say I don’t know if unsure”) barely moves the needle, in the low single digits to fifteen percent, because you have asked the model to be honest about knowledge it still does not have. The instruction is not the fix. The retrieved text is the fix. The instruction just tells the model to lean on it.
Citations you can actually check
Grounding decides what the model may say. Citations are how a human confirms it obeyed. You ask the model to attach, to each claim, which source it came from, and you render those as links straight back to the chunk or document. Now a reader is one click from the evidence, and so is the on-call engineer at 2am trying to work out why the bot said something odd.
Providers have started returning this structurally. Instead of trusting the model to type [2] in the right place, a citations API hands back the span of generated text along with the exact source location it maps to: character offsets, a page, a block id. You can also do it yourself with numbered [n] markers and the retrieved chunks you kept around. Either way, the citation is only worth something if it survives a click. Which brings us to the two ways citations fail.
The first failure is the loud one: a claim with no citation. That is a raw hallucination, and it is the easy case, because it is detectable. Any claim the model could not attach to a source is a claim you can strip, flag, or refuse on.
The second failure is quieter and more dangerous: a citation that exists but does not hold up. The model writes “refunds within 30 days [5]” and source 5 is a page about two-factor auth. The number is there, it links to something real, it looks checked. It is not. A citation that resolves to the wrong passage is arguably worse than no citation, because it manufactures the appearance of verification. This is why you never trust that a citation is correct just because it is present. If the source is text, verify that the cited words actually occur in the cited chunk. That check is cheap and it catches a whole class of confident nonsense.
Faithfulness is not the same as correctness
Two different questions hide inside “is this answer good”, and keeping them apart will save you a lot of confused debugging.
Faithfulness asks: does the answer follow from the source it cites? Correctness (or factuality) asks: is the answer true in the real world? They come apart in both directions. An answer can be perfectly faithful to a source that is itself wrong, in which case it is faithfully, confidently false. And an answer can be true in the world while citing a source that does not actually support it, in which case it got lucky and you have no way to know that.
RAG targets faithfulness on purpose. That is not a compromise, it is the only half you can engineer. Correctness depends on your corpus being right, which is a content and data-freshness problem, the same one that made the RAG pipeline’s refund bot serve a stale answer. Faithfulness depends on the model sticking to whatever corpus you gave it, which is a problem you can prompt, measure, and gate. Get faithfulness solid and correctness reduces to “keep your sources good”, which is at least a problem a human team can own.
Techniques that raise groundedness
Beyond the basic “answer only from context” instruction, a few patterns reliably push more of your answers into that top-right cell. None is magic. Stacked, they add up.
Quote first, then answer. Make the model extract the exact supporting sentence from the context before it writes anything in its own words. This is a structural trick: forcing it to surface the evidence first means it cannot answer a question the context does not actually address, because there will be no quote to pull. Structured output makes this clean, since you can demand a shape that has nowhere to put an unsupported claim.
// Force evidence before prose. If `quote` can't be filled from the
// context, the model has to leave it empty, which surfaces the gap.
const schema = {
type: "object",
properties: {
supported: { type: "boolean" },
quote: { type: "string", description: "verbatim sentence from context, or empty" },
source: { type: "string", description: "the [n] the quote came from" },
answer: { type: "string" },
},
required: ["supported", "quote", "source", "answer"],
};One citation per sentence. Asking for a source on every sentence, rather than one at the end, makes unsupported sentences stick out. A sentence that arrives without a [n] is a sentence to be suspicious of, and now it is visible instead of buried in a paragraph.
A second pass that checks the first. After the model answers, run a separate verification step: split the answer into individual claims, and for each one ask a model (or a smaller, cheaper one) whether the cited source actually supports it. Keep the supported claims. Drop or flag the rest. Yes, it is a second call and it costs latency and tokens. For anything a customer reads as fact, it is usually worth it.
Here is the verifier as code. It is deliberately boring, which is the point: it is just another model call whose only job is to say yes or no per claim.
async function verifyClaims(answer, chunks) {
const claims = splitIntoClaims(answer); // one sentence-ish unit each
const checked = await Promise.all(
claims.map(async (claim) => {
const source = chunks[claim.citedIndex];
const supported = await isSupported(claim.text, source?.text ?? "");
return { ...claim, supported };
}),
);
const clean = checked.filter((c) => c.supported);
const dropped = checked.filter((c) => !c.supported);
return { clean, dropped }; // ship `clean`, log `dropped` as a groundedness signal
}Measure it, don’t eyeball it
You cannot improve what you only notice when a customer complains. Groundedness is measurable, and the metric is not exotic. Take an answer, split it into claims, count how many are supported by the retrieved context, and divide. That ratio is your faithfulness score. One perfectly supported answer scores 1.0. An answer where half the claims are unsupported scores 0.5, and that number should scare you.
You do this at scale the same way you do any eval: assemble a small set of real questions with known-good sources, run your system over them, and have a judge (a model prompted to be a strict claim-checker, spot-audited by a human) compute faithfulness per answer. Track it as a number over time. Then a prompt tweak or a model swap that quietly makes the bot bolder shows up as a faithfulness regression in your dashboard instead of as a support escalation three weeks later. Log the retrieved chunks and the dropped claims alongside every answer, the way tracing AI systems covers, so that when the score dips you can see exactly which claim went unsupported and why.
Where grounding stops
Be honest with yourself about the ceiling. Grounding reduces hallucination hard, but it does not remove it. The model can still misread a passage, over-generalise from it, blend two chunks into a claim neither one made, or quietly fall back on its training when the retrieved context is thin or ambiguous. Every one of those produces a wrong answer with a citation attached. Grounding shrinks the failure rate and makes the failures traceable. It does not get you to zero, and any vendor who tells you their system does not hallucinate is selling you the confident-guess behaviour from the top of this article.
So the last layer is not technical. Anything where a wrong answer has real consequences, money, medical, legal, anything a person will act on, keeps a human in the loop. And you never present a generated answer as authoritative on its own. Show the sources next to it, every time, so the reader can do the one thing the model cannot: check. The failure that pulls AI features from production is not the model being wrong occasionally. It is people trusting a wrong answer because nothing on screen invited them to doubt it. The citation is not decoration. It is the invitation to verify, and it is the whole point.
See grounding and refusal in action
Below is a tiny grounded reader, fully simulated in the page, no model and no network. There is a short passage and a few questions. In grounded mode it answers only from sentences that are actually in the passage, highlights the supporting sentence as its citation, and refuses when the passage says nothing. Flip to ungrounded mode and the same system pads its answers with a plausible extra fact that is not in the passage, drawn in red as unsupported, and it will happily invent a whole answer to the question the passage cannot support. Same questions, two very different levels of trust.
Summary
- A model predicts plausible next tokens. It has no built-in step that checks a fact or notices it does not know, so a confident falsehood and a confident truth are the same kind of output to it.
- Training and benchmarking make it worse by scoring “I don’t know” as wrong, which teaches models to guess boldly rather than abstain.
- Grounding constrains the answer to retrieved sources and tells the model to refuse when they do not cover the question. The retrieved text does the work; the instruction just points the model at it.
- Refusal is a feature. Give the model an explicit, detectable exit string so “I could not find that” beats invention.
- Citations let a human verify. Watch for two failures: a claim with no citation (a raw hallucination) and a citation that resolves to the wrong passage (worse, because it fakes verification). If the source is text, check the cited words actually appear.
- Faithfulness (follows from the source) is not correctness (true in the world). RAG targets faithfulness because it is the half you can engineer and measure; correctness then reduces to keeping your sources good.
- Raise groundedness with quote-then-answer, a citation per sentence, and a second pass that drops unsupported claims before they ship.
- Measure faithfulness as supported-claims over total-claims and track it, so a regression shows up on a dashboard, not in a support queue.
- Grounding reduces hallucination, it does not eliminate it. High-stakes answers keep a human in the loop, and you never show a generated answer as authoritative without its sources beside it.