Cosine Similarity and Nearest Neighbours
A ranking bug once ate an afternoon and taught me more vector math than any tutorial had. I had semantic search running over a few thousand support docs, cosine similarity in the hot loop, working fine. Someone asked me to make search faster. So I swapped cosine for a raw dot product, because the vectors were “basically normalized anyway” and a dot product skips two square roots. Ship it.
Within a day, one document was landing in the top three results for almost every query, including queries it had nothing to do with. It was a giant “all our policies, concatenated” page somebody had pasted together out of forty other pages. Its embedding vector had a bigger magnitude than everything else in the store, and a raw dot product rewards magnitude. Cosine had been quietly dividing that magnitude out the whole time. The instant I dropped the normalization, the longest vector won by default.
That is the entire subject of this lesson in miniature. Embeddings turn text into points in space, and “find related text” becomes “find nearby points.” But “nearby” is not one thing. There are a few different numbers you can compute between two vectors, they disagree in specific ways, and picking the wrong one gives you results that look plausible and are subtly wrong. Then, once you have millions of points, even the right number is too slow to compute against all of them, and you need a smarter way to find the neighbours than checking every single one.
Three numbers between two vectors
You have two vectors and you want a single score for how related they are. Three measures show up in practice, and they are all a few lines of arithmetic.
The dot product multiplies the two vectors component by component and adds it all up. It is the cheapest to compute and it blends two things at once: how aligned the vectors are, and how long they are.
function dot(a, b) {
let s = 0;
for (let i = 0; i < a.length; i++) s += a[i] * b[i];
return s;
}Euclidean distance (also called L2) is the straight-line gap between the two points, the thing you got taught in school as the distance formula. Small distance means close.
function euclidean(a, b) {
let s = 0;
for (let i = 0; i < a.length; i++) {
const d = a[i] - b[i];
s += d * d;
}
return Math.sqrt(s);
}Cosine similarity is the cosine of the angle between the two vectors. It throws away length entirely and looks only at direction. You saw it built from scratch in embeddings; it is the dot product divided by both vectors’ lengths, which cancels the magnitudes out. Two vectors pointing the same way score close to 1, perpendicular scores 0, pointing opposite ways scores close to -1.
Why is cosine the default for text embeddings? Because direction is where the meaning lives, and length mostly carries stuff you do not care about, like how many words were in the passage. A one-line note and a five-paragraph essay about the same bug point the same way in vector space; the essay’s vector is just longer. Cosine treats them as equally relevant, which is what you want. That is the whole reason it dominates for semantic search.
The length trap
The metrics agree a lot of the time, which is exactly why the times they disagree catch you off guard. They part ways over one thing: magnitude.
Look at the left panel. The query and the document sit on the same ray from the origin, so they point in exactly the same direction. Cosine says they are identical, score 1.00, because the angle between them is zero. Euclidean says they are far apart, because one is way out past the other. That is precisely the shape of my policy-page bug. A raw dot product leans the same way Euclidean does here: it grows with magnitude, so the longest vector gets an unfair boost on every query.
The right panel is the reassuring case. Two vectors of the same length, thirty degrees apart. Now cosine and Euclidean rank things the same way, because with lengths held equal, a wider angle is the only thing left that can push points apart. Which points at the fix.
Normalize once, and the three line up
Scale every vector to length 1 before you store it. This is called normalizing, and it is one loop.
function normalize(v) {
let n = 0;
for (const x of v) n += x * x;
n = Math.sqrt(n);
return n === 0 ? v : v.map((x) => x / n);
}Once your vectors are unit length, the three metrics stop disagreeing. The dot product becomes the cosine (the division by lengths is now a division by 1). And Euclidean distance becomes a fixed function of the cosine:
const a = normalize(rawA);
const b = normalize(rawB);
dot(a, b); // now equal to cosine(a, b)
euclidean(a, b); // now equal to Math.sqrt(2 - 2 * dot(a, b))That second line is worth sitting with. sqrt(2 - 2 * cosine) shrinks as cosine grows, with no bumps. So the vector with the highest cosine is always the vector with the smallest Euclidean distance. They produce the exact same ranking. Which means on normalized vectors you get to pick whichever is cheapest to compute (usually the plain dot product) and the results are identical to cosine. This is why so much production code looks like it is “just doing a dot product.” The normalization happened earlier, offline, once.
When you move this into a database, the metric is not an afterthought, it is baked into the index. Postgres with the pgvector extension (v0.8.x as of 2026) gives each metric its own operator and its own operator class, and the two have to match:
-- pgvector distance operators (smaller result means closer)
-- <=> cosine distance (1 - cosine similarity)
-- <-> Euclidean / L2 distance
-- <#> negative inner product (dot product, negated)
-- build the index with the operator class for your metric
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);
-- then query with the matching operator; $1 is the query vector
SELECT id, text
FROM docs
ORDER BY embedding <=> $1
LIMIT 5;Two small gotchas hide in there. These operators return distance, not similarity, so smaller is closer and you sort ascending. And the inner-product operator is negated (<#> gives you minus the dot product) precisely so that “smaller is closer” stays true, since a bigger dot product means more similar. If you build the index with one operator class and query with a different operator, the index gets ignored and you silently fall back to a slow scan.
Before we go faster, play with the geometry. Edit the two vectors below and watch the dot product, the lengths, and the cosine update live, with the actual angle drawn between them. Then flip to the ranking mode to see nearest-neighbour selection: one query, three documents, sorted by cosine.
Nearest neighbours, the honest way
k-nearest-neighbours is exactly what it sounds like. You have a query vector, you have a pile of stored vectors, and you want the k closest ones. The direct way is the one you already know from embeddings: score the query against every stored vector, then keep the best few.
function nearest(query, index, k = 5) {
const scored = index.map((row) => ({
id: row.id,
score: cosine(query, row.vec),
}));
scored.sort((a, b) => b.score - a.score);
return scored.slice(0, k);
}This is correct, it is easy to reason about, and you should reach for it first. For a few thousand vectors it runs in well under a millisecond and there is nothing to tune. Note one small waste: it sorts all n results just to keep k of them. You do not need the tail sorted, so a bounded structure (a small heap of size k) turns the O(n log n) sort into O(n log k). Worth doing, but it is not the part that hurts.
The part that hurts is the map. Every query scores against every row. That is O(n) in the number of documents, and it is called a full scan or a flat search, because you are walking the whole flat list of vectors every single time.
Why the scan falls over
Do the arithmetic and it gets uncomfortable fast. Say you have a million documents at 1024 dimensions each. One query does roughly a million dot products, and each dot product is 1024 multiply-adds. That is about a billion floating-point operations per query, before you have sorted anything. A modern CPU core can push through that, but you are looking at tens of milliseconds per query on a good day, and it scales linearly. Ten million documents is ten times slower. Now serve a hundred queries a second and the maths stops working. You cannot buy your way out with a bigger box forever.
Approximate nearest neighbours
The escape hatch is to stop insisting on the exact answer. An approximate nearest neighbour index (ANN) finds you the closest vectors almost always, by cleverly skipping the vast majority of the comparisons. You trade a sliver of accuracy for an enormous jump in speed, often from a linear scan down to something that behaves like the logarithm of your corpus size.
“Almost always” deserves an honest number. The measure is recall: of the true k nearest neighbours, what fraction did the index actually return. A well-tuned index sits around 0.95 to 0.99 recall, which means every so often it hands back a neighbour that is close but not quite the closest, or misses one genuine top-k hit. For search and RAG that is almost always a fine trade, because the model reading the results shrugs off one slightly-off passage. For something like exact deduplication or a legal lookup where a miss is unacceptable, it may not be, and you keep an exact path for those. The word “approximate” is doing real work. Take it seriously.
There are two dominant strategies, and both are ways of not looking at most of your data.
HNSW: hop through a graph
HNSW stands for Hierarchical Navigable Small World, and it is the default in most vector databases as of 2026 because its speed-versus-recall tradeoff is hard to beat. Ignore the intimidating name and picture a graph. Every vector is a node, and each node is wired to a handful of its nearest neighbours. To find what is close to a query, you start at some node and keep hopping to whichever neighbour is closer to the query, like walking downhill. When no neighbour gets you closer, you have arrived.
The “hierarchical” part is the clever bit that keeps those walks short. The graph is built in layers. The bottom layer holds every vector with dense, short-range links. Each layer above it holds fewer nodes with longer links, like an express highway laid over local streets. A search starts at the top, sprints across the sparse highway to get into the right region fast, then drops down a layer and refines, and drops again, until it is doing fine-grained hops on the bottom layer near the answer.
The knobs you tune trade recall against speed and memory. Bigger neighbour lists and a wider search beam raise recall and cost more time. In pgvector these surface as m and ef_construction when you build the index, and ef_search at query time: turn ef_search up and you look at more candidates, get better recall, and pay for it in latency. The details of running this live in vector search with pgvector.
IVF: cluster first, then search a few clusters
The other common approach flips the problem around. Before any queries arrive, group all your vectors into clusters (typically with k-means), and remember each cluster’s centroid, its average position. That is the “inverted file” idea, IVF. At query time you do not compare against every vector. You compare the query against the handful of centroids, pick the few clusters whose centroids are closest, and only search inside those.
The parameter here is how many clusters you probe (nprobe in most tools, lists and probes in pgvector). Probe more clusters and you catch more true neighbours at the cost of speed. IVF is cheaper to build than HNSW and uses less memory, but at the same recall it is usually a bit slower to query. It also has a catch HNSW does not: it needs a representative sample of your data up front to learn the clusters, so you cannot build a good IVF index on an empty or tiny table.
The curse of dimensionality
There is a reason all of this is engineered so carefully, and it is genuinely weird. In very high-dimensional space, distances stop being as meaningful as your 2D intuition expects. As you add dimensions, the distance from a random point to its nearest neighbour and its farthest neighbour drift closer and closer together, until almost everything is roughly the same distance from everything else. The gap that makes “nearest” mean something narrows toward nothing. People call this distance concentration, and it is the heart of the curse of dimensionality.
So how does any of this work at 1024 dimensions? Because real embeddings are not random points scattered uniformly through the space. A trained model packs meaning onto a much lower-dimensional surface curled up inside the big space (people call it a manifold), and on that surface there is real structure with real neighbours. The curse describes the worst case, uniform random data, and embeddings dodge it by not being random. But the effect never fully goes away, and it is why the last few percent of recall are hard to buy, why quantizing vectors down to fewer bits works better than it has any right to, and why you should be suspicious of ranking decisions made on hair-thin score differences. When two results are 0.9012 and 0.9009 apart, the geometry is not telling you much.
It all lands in the database
You will rarely hand-code HNSW. You pick a metric that matches your embedding model, store your vectors in something that speaks it (Postgres with pgvector, or a dedicated vector database), and let it build the index. The concepts here are what let you read those config options and know what you are trading. Cosine versus inner product is a metric choice. m and ef_search are recall-versus-latency dials. nprobe is the same dial for IVF. The vector search with pgvector lesson wires this into a real schema and query, and once retrieval is fast and correct, it becomes the front half of the RAG pipeline.
The mental model to keep: closeness is a number you choose, not a fact handed to you, and finding the closest vectors quickly is a solved problem as long as you accept “almost always” instead of “always.”
Summary
- Between two vectors you can compute a dot product (blends direction and length), Euclidean distance (straight-line gap, grows with length), or cosine similarity (pure angle, ignores length). Cosine is the default for text because direction carries the meaning.
- The metrics disagree exactly when magnitudes differ. A raw dot product rewards big vectors, which is a classic ranking bug on un-normalized embeddings.
- Normalize vectors to length 1 and the three collapse into the same ranking: dot product equals cosine, and Euclidean becomes
sqrt(2 - 2 * cosine). Use whichever your model was trained for. - k-nearest-neighbours by full scan is
O(n)per query. Correct and simple, fine for thousands of vectors, too slow past that. - Approximate nearest neighbour indexes trade a little recall (typically 0.95 to 0.99) for a huge speedup by visiting only a handful of vectors per query.
- HNSW navigates a layered graph, hopping closer each step from a sparse top layer down to the dense base. IVF clusters the vectors and searches only the nearest few clusters. HNSW usually wins on speed-recall; IVF is lighter and cheaper to build.
- High-dimensional distances concentrate, so “nearest” is fuzzier than intuition suggests. Embeddings survive it by living on a structured lower-dimensional surface, but treat tiny score gaps with suspicion.