Hybrid Search and Reranking
A support engineer messaged me because search was “confidently wrong.” A customer had pasted an exact error string into our help search, ERR-4021, the kind of thing you copy straight out of a red error toast. The top result was a friendly onboarding article called “Understanding your plan limits.” Relevant-ish. Also useless. The one document that actually contained the string ERR-4021, a troubleshooting note with the exact fix, was sitting at rank 14, buried under a dozen articles that were vaguely about plans and limits and nothing about that error.
The search was pure vector search, and it was doing exactly what it was built to do. Embeddings turn text into a point in space where nearby means related in meaning. The query embedded as “something about plan limits,” and the model returned the most on-topic prose it could find. The exact-match doc lost because it was not the most thematically central page. It was just the most literally correct one, and the embedding had no way to know that ERR-4021 mattered more than the surrounding words.
That is the failure mode nobody warns you about. Semantic search feels like magic right up until a user searches for an error code, a product SKU, a config key, or a person’s name, and it hands back something plausible and wrong. The fix is not a bigger embedding model. It is to stop choosing between keyword search and semantic search and run both.
Two kinds of search, two kinds of failure
Keyword search and semantic search fail in opposite directions, and that is the whole reason to combine them.
Keyword search matches the actual words. It is precise on exact terms, rare identifiers, jargon, and anything the user typed verbatim. If someone searches ERR-4021, a keyword engine finds the documents containing ERR-4021 and ranks them by how prominent that term is. It is unbeatable when the user already knows the exact word. It is also brittle: search for “my card got declined” and a keyword engine will completely miss a document titled “Payment authorization failed,” because they share no words even though they mean the same thing.
Semantic search matches meaning. It sails past wording differences, so “my card got declined” lands on “Payment authorization failed” without either sharing a token. That same strength is its weakness. It blurs specifics. An exact identifier gets embedded into a fuzzy neighborhood of “things about that topic,” and the literal match drowns under prose that is more on-theme.
The pattern is symmetric enough to feel like a law. When the query is a precise token, keyword wins and semantic drifts. When the query is a paraphrase, semantic wins and keyword whiffs. Real users send both kinds of query at you, often in the same session, so committing to one engine means signing up for one of these failure modes forever.
What BM25 actually rewards
The keyword side of hybrid search is almost always some flavor of BM25. It is worth understanding what it scores, because its instincts are the exact complement of an embedding’s.
BM25 ranks a document for a query out of three ingredients. Term frequency: a document that uses a query word more is probably more about it. Inverse document frequency (IDF): a word that appears in almost every document (like “the” or, in a support corpus, “account”) carries little signal, while a rare word (like ERR-4021) carries a lot. Length normalization: a long document has more room to contain any given word by chance, so BM25 discounts sheer length so a focused short doc is not buried under a rambling long one.
The clever part, and the reason BM25 beat plain word-counting decades ago, is that term frequency saturates. The first time a query word shows up in a document is strong evidence. The second time adds a bit. By the tenth time, another occurrence barely moves the score. Repeating a word a hundred times cannot make a document a hundred times more relevant, and BM25 bakes that ceiling in.
Two knobs control the shape. k1 sets where saturation kicks in (the knee in the curve, usually around 1.2 to 2.0). b controls how hard length normalization bites (usually around 0.75). You rarely touch these. The defaults are good, and the point here is not to tune BM25 but to notice what it fundamentally is: a scorer that cares about which literal words appear and how rare they are, and nothing about what they mean. That is precisely the axis an embedding is blind to.
Merging two rankings
Run both engines and you get two ranked lists of the same documents. Now you have to combine them into one, and this is where the obvious idea quietly breaks.
The obvious idea is to add the scores. Take each document’s BM25 score, add its vector similarity, sort by the sum. It does not work, because the two numbers live on different scales that have nothing to do with each other. BM25 scores are unbounded and depend on your corpus statistics: a great match might score 18, a mediocre one 4. Cosine similarity is pinned between -1 and 1, and for a normalized embedding model your top hits often cluster in a narrow band like 0.82 to 0.91. Add an 18 to a 0.86 and the BM25 number swamps everything. The vector signal contributes essentially nothing.
You might reach for normalization: rescale each list to 0 to 1 with min-max, then add. That is better and still fragile. Min-max is at the mercy of outliers. One freakishly high BM25 score at the top of the list compresses every other result into a thin band near zero, and the fusion collapses back toward “whatever BM25 thought.” The normalized values also shift around depending on how many candidates you pulled, so the same document can get a different fused score just because the candidate set changed. Score fusion can be made to work, but it needs care and tuning you probably do not want to own.
Reciprocal rank fusion
The reliable default is to throw the scores away and fuse on rank position instead. Reciprocal rank fusion (RRF) gives each document a score based only on where it placed in each list, using one formula:
score(doc) = sum over lists of 1 / (k + rank_in_that_list)rank is the document’s 1-based position in a list (1 for the top hit), and k is a constant that damps how much the very top ranks dominate. The community-standard value is k = 60, and it has held up across a lot of systems since the original paper. A document that ranks well in both lists accumulates two solid contributions and floats to the top. A document that tops one list but sinks in the other gets one big contribution and one tiny one, and it loses to the document both engines agreed was good.
That is the whole trick, and it is a few lines of JavaScript:
// Combine several ranked lists of ids into one, by rank position.
// Each list is ordered best-first. k damps the weight of the top ranks.
function reciprocalRankFusion(lists, k = 60) {
const score = new Map();
for (const list of lists) {
list.forEach((id, i) => {
const rank = i + 1; // 1-based position in this list
score.set(id, (score.get(id) ?? 0) + 1 / (k + rank));
});
}
return [...score.entries()]
.sort((a, b) => b[1] - a[1])
.map(([id]) => id);
}Because it only looks at positions, RRF does not care that BM25 scores and cosine scores are incomparable. Rank 1 is rank 1 in any units. It needs no per-corpus tuning, it is stable when the candidate set changes, and it survives one engine producing a weird score distribution. For most teams it is the right first thing to ship, and often the last.
Play with it below. One fixed query, six documents, ranked one way by keyword and another way by semantic. Hit fuse and watch RRF combine the two rankings. Notice the document that was never number one in either list rising to the top, because it was near the top of both. Then flip k to 1 to see the fusion turn top-heavy, where being number one in a single list starts to win again. It is fully simulated in the page, with hand-assigned ranks and no model or network.
The second pass: reranking
Fusion gives you a solid list. Reranking makes it sharp.
Here is the problem it solves. Your first-stage retrieval, both the keyword and the vector halves, is tuned for recall and speed. It has to. Vector search leans on an approximate index that trades a little accuracy for a huge speedup, and keyword search ranks on term statistics that know nothing about the sentence as a whole. So the first stage is good at not missing the right document, and only okay at putting it in the exact right position. You get a decent top 50. The ordering inside that 50 is rough.
A reranker is a heavier model that fixes the ordering. You hand it the query and each candidate document, it reads each query-document pair together, and it emits a relevance score. Sort by that score, keep the top 5, and send those to the language model. The list that goes into your prompt is now precise, not just recalled.
The reason a reranker can do better than the retriever comes down to architecture. Your embedding model is a bi-encoder: it encodes the query into a vector and each document into a vector, completely independently, and compares them with a cheap dot product. That independence is what makes it fast (you embed every document once, offline, and reuse the vectors forever) and it is also what limits it. The query never actually meets the document. A cross-encoder, the classic reranker architecture, does the opposite: it concatenates the query and one document into a single input and runs them through the model together, so every word of the query can attend to every word of the document. It sees the interaction directly. That is a much richer comparison, and it produces noticeably better ordering.
In code the reranker is a swappable call. The provider changes; the shape does not. You give it the query and the candidate texts, it gives you scores, you keep the best few.
async function retrieve(query, { topK = 5 } = {}) {
// Stage 1: two cheap, recall-oriented searches, in parallel.
const [keywordHits, vectorHits] = await Promise.all([
keywordSearch(query, { limit: 50 }), // BM25 / full-text
vectorSearch(query, { limit: 50 }), // ANN over embeddings
]);
// Fuse the two rankings into one shortlist.
const shortlist = reciprocalRankFusion([
keywordHits.map((d) => d.id),
vectorHits.map((d) => d.id),
]).slice(0, 50);
// Stage 2: a cross-encoder reads each (query, doc) pair and scores it.
const docs = await loadDocs(shortlist);
const ranked = await rerank({
query,
documents: docs.map((d) => d.text),
});
// ranked -> [{ index, relevanceScore }], best first. Keep the top few.
return ranked.slice(0, topK).map((r) => docs[r.index]);
}The two first-stage searches run concurrently with Promise.all, so the slower of the two sets your latency, not the sum. If both halves live in Postgres you can even fuse in SQL and skip a round trip, keeping every parameter bound:
-- Two candidate lists in one query, fused by RRF. $1 = query text,
-- $2 = the query embedding. Both lists are capped at 50.
WITH keyword AS (
SELECT id, row_number() OVER (ORDER BY ts_rank_cd(fts, q) DESC) AS rank
FROM docs, websearch_to_tsquery('english', $1) AS q
WHERE fts @@ q
LIMIT 50
),
vector AS (
SELECT id, row_number() OVER (ORDER BY embedding <=> $2) AS rank
FROM docs
ORDER BY embedding <=> $2
LIMIT 50
)
SELECT id, sum(1.0 / (60 + rank)) AS rrf
FROM (SELECT * FROM keyword UNION ALL SELECT * FROM vector) both
GROUP BY id
ORDER BY rrf DESC
LIMIT 50;That gives you the fused shortlist. The reranker still runs in your application code, because it is a separate model call.
Why you cannot just rerank everything
If the cross-encoder is so much better, why not skip the retrieval stage and rerank the whole corpus? Because the thing that makes it better is exactly the thing that makes it slow.
A bi-encoder embedding is precomputable. You embed every document once, when you index it, and store the vector. At query time you embed only the query and do cheap dot products against the stored vectors. The heavy work happened offline, and it is amortized over every future search.
A cross-encoder cannot do that. Its score depends on the query and the document together, so there is nothing to precompute. Every score is a fresh model pass over one query-document pair, at query time, while the user waits. Rerank 50 candidates and that is 50 model passes. Rerank a million-document corpus and that is a million passes per query, and the arithmetic falls off a cliff.
Batching and a GPU pull the per-pair cost down, and hosted rerankers are optimized hard, but no amount of batching rescues a million passes per query. The shortlist is not a nice-to-have. It is the thing that makes reranking possible at all. First-stage retrieval exists to cut the candidate set down to a size the cross-encoder can afford to read closely, and the two-stage split (cheap-and-wide, then expensive-and-narrow) is the whole design.
What it costs, and when it is worth it
Reranking is not free, and it is worth being honest about the bill before you reach for it.
Latency. A rerank pass over 50 candidates typically adds somewhere in the range of 50 to 300 milliseconds as of 2026, depending on the model, whether it is hosted or self-hosted, and how well you batch. That is on top of retrieval. For a chat feature where the model then spends a couple of seconds generating, it disappears into the noise. For an autocomplete-style search-as-you-type box, it can be the difference between snappy and sluggish, and you might rerank a smaller shortlist or skip it.
Money. Hosted rerankers are usually billed per document scored (or per search), so cost scales with your shortlist size, not your corpus. Reranking 50 candidates is cheap next to the generation call that follows. If you are paying for a big model to read the retrieved context anyway, the reranker is a rounding error that makes that expensive context much more likely to be the right context.
Complexity. It is one more model, one more dependency, one more thing with its own latency and failure modes and rate limits. If retrieval falls over, you want sensible fallbacks, and you want it traced so you can see which stage is slow.
So when is it worth it? Add reranking when the quality of the top few results matters and your first stage is returning the right documents but in a mediocre order. That is most serious retrieval-augmented generation. Skip it, or defer it, when a plain hybrid top-k is already good enough, when latency is brutally tight, or before you have measured that ordering is actually your problem. The honest sequencing is: start with vector search, add keyword search and fuse with RRF when exact terms start failing, and add a reranker when you can measure that a better top 5 would meaningfully improve answers.
Where this sits in the pipeline
Hybrid search and reranking are the retrieval half of the RAG pipeline. Everything here feeds the same downstream step: assemble the best handful of chunks and put them in the prompt so the model answers from real, current, permission-scoped text instead of its own memory.
It is tempting to think the way to improve a RAG system is a bigger model or a longer prompt. Usually it is not. A model given twenty mediocre chunks writes a worse, more expensive, more easily distracted answer than the same model given five precise ones. Better retrieval beats a bigger prompt almost every time, and it is cheaper. Fewer tokens in, more relevant tokens, a sharper answer. When you later add citations so the model has to point at its sources, precise retrieval is what makes those citations actually support the claim rather than being loosely on topic.
The mental model to keep: retrieval is a cheap-and-wide first pass followed by an expensive-and-narrow second pass. Keyword and vector each catch what the other drops, RRF merges them without you having to reconcile their scores, and the reranker spends real compute to get the final ordering right on the handful of results that reach the model. Each piece is optional, and you add them in the order your failures demand.
Summary
- Pure vector search blurs specifics. It fails on exact terms (error codes, SKUs, names, code symbols) because embeddings capture meaning, not literal tokens. Keyword search fails on paraphrases for the opposite reason. Run both.
- Keyword ranking is usually BM25: it rewards rare query terms (IDF), saturates term frequency so repetition stops helping, and discounts length. It knows nothing about meaning, which is exactly the axis embeddings are blind to.
- Do not add scores from different engines. BM25 and cosine live on incompatible scales, and min-max normalization is fragile to outliers.
- Reciprocal rank fusion merges lists by rank position with
1 / (k + rank)andk = 60. It ignores raw scores, needs no tuning, and rewards documents that rank well in both lists over ones that win a single list. - Reranking is a second pass. First-stage retrieval is tuned for recall and speed and returns a rough top 50; a cross-encoder reads each query-document pair together and reorders down to a precise top 5.
- A bi-encoder embedding is precomputed offline; a cross-encoder score depends on query and document together, so it runs per pair at query time. That is why you cannot rerank the whole corpus, only the shortlist.
- Reranking adds roughly 50 to 300 ms and a per-document cost, cheap next to the generation call. Add it when a better top 5 measurably improves answers. Recall at stage one is a hard ceiling the reranker cannot lift.
- Hybrid plus rerank is the strong, evergreen default for serious retrieval. Better retrieval beats a bigger prompt: fewer, more relevant chunks produce sharper, cheaper answers.