Chunking: The Part Everyone Gets Wrong
The first document-search bot I put in front of real users could find the right page and still answer wrong. Someone asked how to cancel their plan. Retrieval did its job and pulled back the single most relevant chunk, which read, in full: “Q: How do I cancel my subscription?” That was the whole chunk. The answer, “Open Settings, then Billing, then Cancel,” had been sliced into the next chunk, which scored a hair lower and never made the cut. So the model received a question with no answer attached, and did what models do when you hand them a blank: it made up a cancellation flow that did not exist.
Nobody had touched the retriever. The embeddings were fine. The nearest-neighbour search was fine. The bug was one step earlier, in how the document got chopped into pieces before any of that ran. That step is chunking, and it is the piece of a retrieval system that people spend the least time on and lose the most quality to.
This one is evergreen. Models, embedding APIs and vector databases all churn every few months. How you cut a document into retrievable pieces has barely changed in years, and it is still the highest-impact decision in the whole pipeline.
Why you split documents at all
Two reasons. One is obvious and one bites quietly.
The obvious one: an embedding model reads its input as tokens and has a hard cap on how many it will take. As of 2026 that cap ranges from a few hundred tokens on older models to tens of thousands on newer ones, and it keeps moving. Either way, a forty-page handbook is not going through one embedding call.
The quiet one is the reason that matters even when the whole document does fit. An embedding is a single point in space that stands for the meaning of its input. Embed a whole handbook as one vector and you get the average of forty pages: billing, login, exports, SSO, rate limits, the lot. That average sits in the dead centre of all of it and is sharply close to none of it. A query about “resetting SSO” has to out-shout thirty-nine other topics smeared into the same point, and it loses.
So you split the document into passages, embed each passage on its own, and store each as its own row that points back at its source. Retrieval then finds the passage that answers the query instead of the document that happens to contain it. The rest of this article is about doing the splitting well, because the naive version is a trap.
How blind slicing cuts you
The lazy chunker is three lines: walk the string, cut every N characters, done. It runs fast and it is where most RAG quality quietly dies, because a character count knows nothing about where meaning starts and stops. It will slice a sentence mid-word. It will sever a question from its answer, which is the bug I opened with. It will cut a table loose from its header row so the numbers lose their columns. It will split a function so neither half parses.
The thing to internalise: a chunk is retrieved and read alone. The model never sees it in the flow of the surrounding document. Whatever context a human would use the neighbouring paragraphs to supply, the chunk has to carry itself, or it is gone. That single constraint drives every decision below.
The three levers you actually control
Strip away the jargon and there are three dials: how big each chunk is, how much adjacent chunks overlap, and where you allow the cuts to fall. Get these three right and you are most of the way there.
Size: the dial between starved and diluted
Chunk size trades two failures against each other.
Too small and each chunk is a fragment with no context. The sentence “then click Cancel” retrieves perfectly for a query about cancelling, but on its own it does not say cancel what, or where the button is. You matched the needle and threw away the haystack it needed.
Too large and you are back to the blurry-average problem from the top of the page, just at a smaller scale. A 3,000-token chunk covering six subtopics gets one vector that is the centroid of all six. It matches every one of them weakly and none of them sharply. It also eats your context window: stuff five of those into a prompt and you have burned 15,000 tokens to deliver maybe 2,000 tokens of relevance.
Treat any specific number as a starting point, not a law. Shorter chunks suit short factual lookups where the answer is one line. Longer chunks suit questions that need a few paragraphs of reasoning to answer. The only way to know what your corpus wants is to try a couple of sizes and measure retrieval, which is the last section of this article.
Overlap: insurance for facts on the boundary
Cut a document into back-to-back chunks and some unlucky fact lands right on a seam. “The migration deadline is April 30” gets split so “the migration deadline is” ends one chunk and “April 30” starts the next, and now no single chunk contains the whole fact. Neither one retrieves well for “when is the migration deadline.”
Overlap fixes this by letting each chunk repeat the tail end of the one before it. The window slides forward by less than its own width, so the boundary regions appear in two chunks instead of falling between them.
In code the fixed-size version is a two-liner: the window is size wide and advances by size - overlap each step.
function fixedChunks(text, size, overlap) {
const step = size - overlap; // window advances by less than its width
const chunks = [];
for (let i = 0; i < text.length; i += step) {
chunks.push(text.slice(i, i + size));
}
return chunks;
}
// size 800, overlap 100 => each chunk repeats the last 100 chars of the previous oneA common starting point is overlap around 10 to 20 percent of the chunk size. Do not treat it as free, though. Overlap is duplicated text: more chunks, more storage, more embedding cost, and the same passage can now show up twice in your top results. Honestly, the evidence on how much it helps is mixed. Some 2026 benchmarks found a clear benefit, at least one found close to none in its setup. My take: a modest overlap is cheap insurance against boundary-straddling facts, but it is not a substitute for cutting in the right places to begin with, which is the next lever and the important one.
Structure: cut on the document’s own seams
Here is the lever that matters most and the one blind slicing throws away. Documents already tell you where the natural boundaries are. Headings, blank lines between paragraphs, list items, code fences, table rows. Those are the seams a human would cut on, and cutting on them gives you chunks that are coherent units instead of arbitrary spans.
The standard approach is a recursive splitter. Instead of one separator, you give it a hierarchy from coarsest to finest, and it always cuts on the coarsest boundary that keeps a piece under your size limit. Only when a piece is still too big does it drop to a finer separator.
The shape of it is small enough to write yourself and understand end to end:
// Cut on the coarsest separator that keeps a piece under `max`,
// and only fall back to a finer one when a piece is still too big.
const SEPARATORS = ["\n## ", "\n\n", "\n", ". ", " "];
function chunk(text, max, seps = SEPARATORS) {
if (text.length <= max) return [text];
for (const sep of seps) {
if (!text.includes(sep)) continue; // this seam isn't in the text
const finer = seps.slice(seps.indexOf(sep) + 1);
const out = [];
let buf = "";
for (const part of text.split(sep)) {
const next = buf ? buf + sep + part : part;
if (next.length <= max) {
buf = next; // still fits, keep packing
} else {
if (buf) out.push(buf);
// a single part over the limit gets split by a finer separator
buf = part.length <= max ? part : (out.push(...chunk(part, max, finer)), "");
}
}
if (buf) out.push(buf);
return out;
}
return [text.slice(0, max), ...chunk(text.slice(max), max, seps)]; // hard cut
}Real libraries add polish (smarter separators per language, token counting instead of character counting, tidier handling of the overlap), but the idea is exactly this. It is not clever. It just refuses to cut in a stupid place when a sensible one is right there.
Semantic chunking, and whether it earns its cost
There is a fancier idea that sounds obviously better: cut where the topic changes, not where the structure happens to break. Embed each sentence, walk through them in order, and start a new chunk whenever the meaning jumps, measured by the distance between neighbouring sentence vectors. When two adjacent sentences are far apart in vector space, that gap is probably a topic boundary.
It is a genuinely nice idea and it can help, especially on messy text with no reliable structure to lean on: scraped web pages, transcripts, walls of prose with no headings. But it is not a free win, and this is where you have to be a bit skeptical of the hype.
When a clean chunk still loses its thread
Even a perfectly cut chunk can be stranded. Picture a release note whose third paragraph reads “It shipped in June and fixed the timeout.” Cut cleanly on the paragraph boundary, that chunk is coherent English, and it is also useless to retrieve, because it is some feature named two paragraphs up that this chunk no longer contains. The pronoun lost its referent at the cut.
Two techniques address this directly, and both are worth knowing because they are provider-agnostic and here to stay.
The first is to give each chunk a little context before you embed it. You prepend a short sentence that situates the chunk in its document, something like “From the v9 release notes, section on networking:”, generated once per chunk. The chunk’s vector now encodes where it lives, not just its bare words, and retrieval stops confusing similar-looking chunks from different documents. Published results put the drop in retrieval misses in the tens of percent. The cost is one cheap model call per chunk at index time, which you pay once, and which prompt caching makes cheaper still.
The second is late chunking, which needs a long-context embedding model that pools token vectors into a chunk vector. Instead of embedding each chunk in isolation, you run the whole document through the model first so every token vector is computed with the entire document in view, and only then pool the tokens into per-chunk vectors. Each chunk’s vector carries a memory of the surrounding document without any extra model calls. The gains reported are smaller than the prepend-context trick but consistent, and it is an architectural change rather than a per-chunk cost.
Metadata: the cheapest quality you will ever add
A chunk is not just its text. The single highest-return, lowest-effort thing you can do is attach structured metadata to every chunk as you index it, because a bare vector can find a passage but can do nothing else with it.
The record you store per chunk looks something like this, and the extra fields cost you almost nothing to fill in at index time:
const record = {
id: "billing-faq#cancel",
text: "To cancel, open Settings, then Billing, then Cancel plan...",
vector: embedding, // for the nearest-neighbour search
source: "billing-faq.md", // so the answer can cite where it came from
section: "Cancelling a plan", // a human-readable location
updatedAt: "2026-05-01", // for freshness and time filters
tenantId: "acme", // for access control at query time
};Each field buys you something you cannot get from the vector alone. source and section let the model cite where an answer came from instead of asserting it into the void. updatedAt lets you prefer fresh chunks or exclude stale ones. And tenantId is the one that keeps you out of the news.
The filtered query is just a normal parameterised statement with a vector distance in the ORDER BY:
-- only this tenant's fresh chunks, ranked by cosine distance to the query vector
SELECT id, source, section, text
FROM chunks
WHERE tenant_id = $1
AND updated_at >= $2
ORDER BY embedding <=> $3
LIMIT $4;One splitter does not fit every document
Prose is the easy case. The moment your corpus has code, tables or transcripts in it, a single strategy tuned for paragraphs starts producing garbage on everything that is not a paragraph. Match the cut to the shape of the content.
The table rule is the one people miss most, and it is the most damaging. A markdown table has its column names in the first row and nowhere else. Slice it into row-range chunks the naive way and every chunk except the first is a grid of numbers with no idea what the columns mean. “94, 12, 3” retrieves for nothing and answers nothing. Repeat the header row into each chunk and suddenly “region: EU, signups: 94, refunds: 12, churn: 3” is a fact a model can actually use.
A pattern that scales when your inputs are mixed is to route by type: detect what a document is, then send it to the splitter built for that shape. Code files to a code-aware splitter, spreadsheets and tables to a row splitter that carries the header, everything else to your recursive prose splitter. One size does not fit all, and pretending it does is how half your retrieval quietly turns to noise.
See it move
Numbers on a page do not build intuition. Drag the sliders below and watch chunks form over a sample document. Toggle between blind fixed-size slicing and structure-aware splitting, and watch the fixed-size mode cut straight through words while the structure mode keeps paragraphs whole. Turn overlap up in fixed mode to see the repeated regions light up.
Notice two things as you play. In fixed mode the coloured boundaries fall wherever the character count runs out, straight through “pass|word” and “Set|tings,” and the amber overlap regions are the price you pay to stop facts falling between chunks. Switch to paragraph mode and the boundaries snap to the blank lines between sections, so each chunk is a whole heading-plus-answer. Same document, same size budget, completely different retrievability.
Chunking is a loop, not a setting
The mistake underneath all the others is treating chunking as a config value you set once and forget. It is not. It is a parameter you tune against your real corpus and your real queries, and the only way to tune it is to measure.
The measurement is retrieval quality, and it is refreshingly concrete. Take a set of real questions. For each, note which chunk should come back. Run retrieval and check whether that chunk actually lands in the top handful of results. The fraction of questions where the right chunk shows up in the top k is your recall, and it is the number chunking moves. Change your size, your overlap or your strategy, re-index, and watch that number go up or down. This is exactly the kind of thing evals exist to automate, and retrieval recall is one of the easiest and most valuable evals you will ever write.
There is no universal right answer here, and anyone who quotes you one exact chunk size for all documents is selling something. What holds up across corpora is the shape of the advice, not the numbers: structure-aware chunks, sized to hold one coherent idea, with a modest overlap and good metadata, beat blind fixed-size slicing almost every time. Start there, measure, and let your own evals talk you into anything fancier.
Summary
- You chunk because embedding models cap their input, and more importantly because one vector for a whole document is a blurry average that matches nothing sharply. Split into passages, embed each, and retrieve the passage that answers the query.
- Blind fixed-size slicing cuts mid-sentence, severs questions from answers, and strips tables of their headers. A retrieved chunk is read alone, so it has to make sense alone.
- The three levers are size, overlap and where the cuts fall. Size trades a starved fragment against a diluted blur; overlap is cheap insurance for boundary-straddling facts; cutting on the document’s own seams is the lever that matters most.
- A recursive splitter cuts on the coarsest natural boundary that fits and only goes finer when forced, which keeps chunks coherent for a few lines of code.
- Semantic chunking can help on unstructured text but costs an embedding per sentence, and 2026 benchmarks disagree on whether it beats plain structure-aware splitting. Start structure-aware and measure before upgrading.
- Contextual chunking (prepend a situating sentence before embedding) and late chunking (embed the whole document, then pool per chunk) both fix chunks that lost their document context. They are refinements, not day-one work.
- Metadata is the cheapest quality you can add: source and section for citations, dates for freshness, tenant and permissions filtered during retrieval, never after.
- Match the splitter to the content: paragraphs for prose, function boundaries for code, header-carrying rows for tables, speaker turns for transcripts.
- Chunking is a loop. Measure retrieval recall against real questions, adjust, re-index, repeat. When a RAG answer is wrong, read the retrieved chunks before you touch the model.