Evals: Testing Something Non-Deterministic
We swapped the model behind a support-reply drafter to shave the bill. The cheaper one was faster, passed every manual check I threw at it, and read fine in the three tickets I opened to sanity-check the change. It shipped on a Thursday. The following week a team lead asked why refund replies had started quoting the wrong policy window, and only for one category of ticket. Nothing in the code had changed except one line: the model name.
Here is the part that stung. There was no way to catch it. I had no test that would have gone red, because the output was different every run and “different” is not the same as “wrong”. My whole safety net was me, reading a handful of examples and nodding. That works right up until the day it silently doesn’t, and you find out from a customer.
That gap is what evals fill. Not a fancier unit test. A different tool for a different problem: measuring the quality of a system whose output you cannot predict character for character, so that a change which makes things worse fails your pipeline instead of your users.
Why assert-equal falls apart
A unit test rests on one assumption you have never had to question: the same input produces the same output, and you know what it should be. sum(2, 2) is 4. Forever. You write expect(sum(2, 2)).toBe(4) and walk away.
Point that at a model and three things break at once.
First, the output is not stable. The model hands you a probability distribution and a sampler draws from it, so the same prompt can return “You have 30 days to request a refund” one run and “Refunds are available within thirty days of purchase” the next. Both are correct. Neither equals the other. If you want the mechanism, it is in temperature and sampling, and even at temperature zero you are not guaranteed identical bytes across model versions or hardware.
Second, there is rarely one right answer. Summaries, replies, explanations, extracted fields phrased three valid ways: the space of acceptable outputs is large and fuzzy. toBe wants a single string. You do not have one.
// This "test" is a coin flip. It fails on a perfectly good answer
// and tells you nothing about quality.
const reply = await draftReply(ticket);
expect(reply).toBe("You have 30 days to request a refund."); // brittle nonsenseThird, and this is the quiet killer, manual spot-checking does not scale and does not regress. “Looks good in my three tries” is not a measurement. It is a vibe. It cannot tell you that your new prompt is two points better on clarity but four points worse on staying grounded in the docs. It cannot re-run itself on every pull request. It is exactly how a regression like my refund bug reaches production and then hides, because nobody re-reads the same thirty examples every week looking for the one that drifted.
What an eval actually is
Strip away the tooling and an eval is three things.
- A dataset: representative inputs, ideally paired with an expected answer or a rubric describing what “good” means.
- A runner: code that feeds each input through your system exactly as production would.
- Graders: functions that score each output, so the whole run collapses to a number you can track over time.
That last word matters. The output of an eval is not pass or fail, it is a score. A percentage, an average rubric rating, a pass rate on a set of checks. Something you can put on a chart and watch across versions of your prompt, your model, and your retrieval.
In code, the runner is almost embarrassingly plain. It is a loop.
async function runEval(dataset, system, graders) {
const rows = [];
for (const item of dataset) {
const output = await system(item.input); // your real pipeline
const scores = graders.map((g) => g(output, item)); // each grader returns 0..1
rows.push({ item, output, scores });
}
const flat = rows.flatMap((r) => r.scores);
const passRate = flat.filter((s) => s >= 1).length / flat.length;
return { rows, passRate }; // the number you chart
}The interesting engineering is not the loop. It is the two things the loop calls: what goes in the dataset, and how the graders decide what “good” means. The rest of this article is those two problems.
Graders: from cheap and certain to broad and fuzzy
There is no single way to score a model’s output, and the choices sit on a spectrum. At one end, cheap checks that are dead certain but only work when there is a hard fact to check. At the other, a second model judging free-form quality, which covers everything but is fuzzy and needs babysitting. Pick the strongest grader the task will allow, and drop down the ladder only when the output stops having a checkable structure.
Structural checks: free and certain
If any part of the output has a shape, check the shape. Did it return valid JSON? Does that JSON match the schema? Did the agent call the tool you expected with arguments in range? Is the returned total a positive number? These graders are ordinary code. No model, no tokens, no flakiness. They run in a millisecond and they never lie to you.
import { z } from "zod";
const Invoice = z.object({
vendor: z.string().min(1),
total: z.number().positive(),
currency: z.enum(["USD", "EUR", "GBP"]),
});
function structuralGrader(output) {
const parsed = Invoice.safeParse(output);
return parsed.success ? 1 : 0; // no argument about what "valid" means
}You will get more mileage out of these than you expect, because a shocking share of “the AI is broken” incidents are really “the AI returned malformed JSON and my parser threw”. If you are asking the model for structured data, lean on structured output to constrain the shape, then let a structural grader hold it to the schema on every run. When a hard check can answer your question, nothing higher on the ladder is worth the cost.
Reference metrics: comparing to a known-good answer
Sometimes you have a reference answer and want to know how close the output is. The naive version is exact string match, and you already know why that fails: “Canberra” and “The capital is Canberra” are both right and share zero equality.
Softer comparisons help a little. Keyword or token overlap (older text metrics like BLEU and ROUGE are variations on this) reward outputs that share words with the reference. They are fast and cheap. They also miss that “thirty days” and “30 days” mean the same thing while happily passing an answer that reuses your keywords to say something false. Embedding similarity is a step up: encode both strings as vectors and take the cosine, so paraphrases score high even with different words. It is the same nearest-neighbour math from embeddings, just applied to grading.
// Cosine similarity between the output and a reference answer.
function cosine(a, b) {
let dot = 0, na = 0, nb = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i];
na += a[i] * a[i];
nb += b[i] * b[i];
}
return dot / (Math.sqrt(na) * Math.sqrt(nb)); // 1 = identical direction
}
function similarityGrader(outputVec, referenceVec, threshold = 0.82) {
return cosine(outputVec, referenceVec) >= threshold ? 1 : 0;
}Reference metrics have a hard limit. They measure resemblance to one answer you wrote down, not correctness, and definitely not whether the tone is right or the reasoning holds. For anything open-ended they run out of road, which is where the last rung comes in.
LLM-as-judge: a model grading the model
When the thing you care about has no schema and no single reference (is this summary faithful, is this reply helpful and on-policy, is this explanation actually correct), you can ask another model to score it against a written rubric. This is LLM-as-judge, and it is the only grader that scales to open-ended quality without a human reading every output.
const rubric = `Score the support reply from 0 to 1.
1.0 correct policy, cites the right window, polite
0.5 correct but missing a detail or slightly off tone
0.0 wrong policy, invents facts, or rude
Return only JSON: {"score": <number>, "reason": "<short>"}`;
async function judgeGrader(output, item) {
const verdict = await model.json({
system: rubric,
user: `Question: ${item.input}\nReply to grade: ${output}`,
});
return verdict.score; // 0, 0.5, or 1
}This works, and it is genuinely useful. It is also a model, which means it inherits every failure mode a model has, now pointed at grading. Be honest with yourself about that. A judge is a system under test, not an oracle.
The fix for a fallible judge is the same fix you would use for any component you do not fully trust: measure it. Take a few hundred outputs, have a human label them, and check how often the judge agrees. Cohen’s kappa is the standard number here because it corrects for agreement you would get by chance. If the judge and your humans agree at a decent clip, you can let it run at scale. If they do not, your rubric is broken and every green number it produces is a lie.
The playground below makes the grader spectrum concrete. Five question-and-answer pairs, each with a reference answer and a candidate the model produced. Flip between grading methods and watch not just the pass rate move, but which cases each method gets wrong compared to a human label.
Notice the trap. Exact match scores 20 percent and is obviously broken. Keyword overlap scores 80 percent, which looks healthy, but it agrees with the human labels only three times out of five: it passes a confidently wrong answer and fails a correct one. Same headline number, completely different trustworthiness. The pass rate alone can fool you. Agreement with a human ground truth is the number that tells you whether your grader is any good.
Building the dataset
A grader is only as useful as the inputs you feed it. A dataset of ten happy-path questions will happily report 100 percent while your product falls over on everything real users actually send.
Start with the traffic you already have. Sample real production inputs, strip anything sensitive (the PII and redaction rules apply here too, evals are not exempt), and skew the sample toward the messy stuff: the long inputs, the ambiguous ones, the ones in the second language your users speak. Then deliberately add cases you know are hard, plus adversarial inputs designed to make the system misbehave.
The rule that compounds over time: every bug that escapes becomes a permanent test case. My refund regression should have ended with that exact ticket, its expected answer, and a grader added to the suite forever. That is how the dataset earns its keep. It is not a fixed artifact you write once, it is a growing memory of every way your system has ever been wrong, so it can never be wrong that way again without you knowing.
Different tasks need different evals
One overall score hides where things broke. A RAG pipeline that scores badly could be failing at retrieval (it never found the right document) or at generation (it found the right document and then ignored it). Those are different bugs with different fixes, and a single number cannot tell them apart. So you split the metric by stage.
For RAG, grade the two halves apart. On the retrieval side, did the right documents come back at all? Context recall asks whether everything you needed was retrieved, context precision asks how much of what came back was actually relevant. On the answer side, faithfulness (also called groundedness) asks whether every claim in the answer is supported by the retrieved context, and answer relevancy asks whether it actually addressed the question. The diagnostic power is in the combination. High faithfulness with low relevancy is a classic signature: the model wrote a perfectly grounded answer about the wrong thing, which means your retrieval fetched the wrong context. Fix the retriever, not the prompt. If you are also asking the model to cite sources, grounding and citations gives you a structural grader almost for free: check that every citation points to a chunk that was actually retrieved.
For agents, the top-line metric is task completion, and the honest way to measure it is on the end state, not the transcript. Did the calendar event actually get created? Is the row in the database? An agent that says “Done!” without doing the thing should score zero. Beyond completion, grade the trajectory: how many steps did it take, did it call the right tools with valid arguments (an easy structural check against your tool catalog), did it loop or retry sanely. An agent that reaches the right answer in fifteen steps when five would do is not passing. It is a latency and cost regression wearing a green checkmark, and step count is what catches it. The agent loop is where those steps come from.
Running evals in CI
An eval you run by hand when you remember to is barely better than spot-checking. The discipline only kicks in when the machine runs it for you, on every change, and blocks the merge when the score drops. This is the same instinct as CI/CD for ordinary code, applied to prompts, models, and retrieval.
The gate is a threshold. Compute the score, compare it to a bar (or to the current main branch score), exit non-zero if it dropped. That is the whole trick, and it turns a fuzzy quality question into a build status.
// eval.test.js runs in CI like any other test
const BASELINE = 0.8; // hold the line, or read main's score from an artifact
const { passRate, rows } = await runEval(dataset, draftReply, graders);
if (passRate < BASELINE) {
const failures = rows.filter((r) => r.scores.some((s) => s < 1));
console.error(`Eval regressed: ${(passRate * 100).toFixed(0)}% < ${BASELINE * 100}%`);
for (const f of failures) console.error(" fail:", f.item.input);
process.exit(1); // red build. the prompt change does not merge.
}You do not have to build the runner yourself. As of 2026 there are open-source eval tools that plug straight into CI, and they broadly come in two flavors: config-first CLIs where you declare cases and assertions in YAML, and test-runner-native libraries where an eval is just a test with a quality assertion. Both split the same two grader kinds you have already met: deterministic checks that run free, and model-graded checks that cost tokens. Which brings up the catch.
Offline gates and online signals
Everything so far is an offline eval: a fixed dataset, run before you ship, answering “is this change safe to merge”. It is necessary and it is not sufficient, because your dataset can only contain problems you already thought of. Production will find new ones.
That is what online signals are for. Once the thing is live, real behavior grades it for you, if you instrument the right events. The most useful signal is usually the cheapest to capture: the retry or regenerate click. When a user asks for the answer again, they are telling you the first one missed, without filling in a survey. Edit distance is another good one, how much the user changed your draft before sending it. A reply they send untouched scored higher than one they rewrote. Repeated queries, escalations to a human, abandoned sessions: all of it is unlabeled grading happening for free.
Offline and online are not rivals, they are a loop. Offline tells you a change is safe to ship. Online tells you whether what you shipped still works in the wild. And the two connect at exactly one point that makes the whole system compound: when an online signal surfaces a new failure, you turn that real example into an offline case with a grader. The bug that production caught becomes the regression test that CI catches next time. Miss that step and you fix the same class of bug forever. Take it, and your eval suite grows into a precise map of everywhere your system has ever been weak.
The mindset shift
The hard part of all this is not the code. The runner is a loop, cosine is four lines, a threshold gate is an if. The hard part is a change in how you treat the moving pieces.
For most of your career a prompt was a string you tweaked until the demo looked good, and the model was a fixed dependency. Evals ask you to treat both as things under test. Your prompt has versions and a score. Your model choice has a score. Your retrieval settings have a score. When you change any of them, you are not “trying something”, you are running an experiment with a measured outcome, and you keep the change only if the number went up. That is the entire difference between a demo that impressed someone once and a product that stays good while five people change it every week.
Summary
- Unit tests assume one predictable answer. Models break that assumption, so
assert-equalflakes on good output and, worse, never catches a silent quality regression. - An eval is three parts: a dataset of representative inputs, a runner that executes your real system, and graders that score the output into a number you track over versions.
- Graders sit on a ladder. Structural checks (valid JSON, right tool, in range) are free and certain but need hard structure. Reference metrics compare to a known answer. LLM-as-judge scores open-ended quality but is itself fallible.
- A judge is a system under test. It has position, verbosity, and self-preference biases. Validate it against human labels and track agreement (Cohen’s kappa) before you trust its scores.
- Grow the dataset from reality. Real traffic, edge cases, adversarial inputs, and above all every escaped bug turned into a permanent case so the same failure cannot return unseen.
- Split the score by stage. RAG needs separate retrieval and answer metrics. Agents need task completion checked on the end state, plus step count and tool-call accuracy.
- Run evals in CI and gate on the score. Cheap deterministic checks on every commit, expensive judge evals on a sample per PR and a full sweep nightly, so a regression fails the build, not the user.
- Offline gates, online signals. Offline says safe to ship; online (retry rate, edit distance) says still working. Feed production failures back into the offline suite so the loop compounds.
- Treat prompts, models, and retrieval as things under test, versioned and measured. Evals are the discipline that separates a demo from a product.