Temperature, Sampling and Determinism
A field extractor I shipped read scanned invoices and returned a tidy little JSON object: vendor, date, total, currency. It passed every test I wrote. Then finance flagged that one invoice, uploaded twice on two different days, had come back with the currency as USD the first time and US$ the second. Same file. Same prompt. Same model version. Nothing in my code had changed between the two runs.
The culprit was one number I had never set. It sat at the provider’s default, and that default was tuned for chatty, human-sounding replies, not for pulling three fields out of a document. I dropped it close to zero, the wobble went away, and the whole feature got boring in exactly the way an extraction pipeline should be boring.
That number is the temperature, and it does not touch the model at all. The model was the same before and after. What changed was how I turned the model’s answer into an actual token. In what a language model actually does you saw that a model does not pick a word. It produces a probability for every token in its vocabulary and leaves the picking to a separate step. This article is about that step: how you turn a distribution into one token, the handful of knobs that control it, and the determinism myth that trips up almost everyone the first time they try to write a test.
The pick is a step you own
Hold the two stages apart, because conflating them is where the confusion starts. Stage one is the model: tokens in, a probability for every possible next token out. That distribution is fixed for a given input and a given model. Stage two is sampling: you take that distribution and choose one token from it. The knobs live entirely in stage two.
The simplest possible sampler ignores the odds beyond first place. It takes the single most likely token, every time. That is called greedy decoding, and it is the most predictable thing you can do:
// The model already handed you `probs`: one probability per candidate token.
function greedy(probs) {
let best = 0;
for (let i = 1; i < probs.length; i++) {
if (probs[i] > probs[best]) best = i;
}
return best; // always the top token, no randomness
}Greedy is repeatable and often a little flat. It will never surprise you, which is exactly what you want for a classifier and exactly what you do not want for a brainstorm. Everything past this point is about deliberately, controllably letting the sampler reach past first place.
Temperature reshapes the odds
Before you sample, you can reshape the distribution to be more or less peaked. That reshaping is temperature, and the mechanism is one division.
Recall that the model’s raw outputs are logits (unbounded scores), and softmax turns them into probabilities. Temperature slips in between: divide every logit by T, then run softmax on the result.
function withTemperature(logits, T) {
if (T === 0) return argmax(logits); // temperature 0 is just greedy
const scaled = logits.map((l) => l / T); // the whole trick: one division
return sampleFrom(softmax(scaled)); // then pick, weighted by the result
}Think about what dividing does. When T is small (say 0.2), you are dividing by a small number, which stretches the gaps between logits wide. The top score pulls even further ahead, softmax piles almost all the probability onto it, and the sampler picks it nearly every time. The distribution gets sharper. When T is large (say 1.5), you divide by a big number, which squashes the logits toward each other. Softmax spreads the probability out, and lower-ranked tokens get a real shot. The distribution gets flatter.
Read the three panels left to right and you have the whole intuition. Low temperature is a focused model that keeps picking its favourite. High temperature is a loose one that wanders down the list. Push temperature high enough and the distribution goes nearly flat, at which point the model is basically choosing tokens at random and the output turns to word salad. There is a ceiling, and past it you get incoherence, not creativity.
Greedy versus sampling, on one prompt
The clearest way to feel the difference is to run the same prompt many times each way. Greedy hands back the identical answer on every run. Sampling draws from the distribution, so you get variety, and occasionally you get the tail.
Which one you want falls straight out of the task. Classifying a ticket into one of five buckets? You want the same answer for the same input, so greedy or near-greedy. Drafting five different subject lines for a marketing test? You want the spread, so you sample with real temperature. The mistake is reaching for the wrong tool: a brainstorm at temperature 0 gives you the same tired idea five times, and an extraction at temperature 1 gives you the US$ bug from the top of this page.
Trimming the tail: top-p and top-k
Temperature reshapes the whole distribution, tail included. Sometimes that tail is the problem. Even after tempering, tens of thousands of junk tokens each carry a tiny probability, and their combined mass is not nothing. Roll the dice enough times and the sampler eventually lands on one of them, which is how you get a fluent sentence that suddenly veers into a genuinely bizarre word. Truncation cuts the tail off before you sample.
There are two classic ways to do it.
Top-k keeps the k most likely tokens and throws the rest away. Pick k = 40 and the model may only ever choose from its top 40 candidates, renormalised to sum to 1. Simple, but blunt: k is a fixed count that ignores how confident the model is. When the model is certain (one token at 0.98), you have dragged in 39 near-zero options you did not need. When it is genuinely torn across hundreds of reasonable tokens, k = 40 amputates most of them.
Top-p, also called nucleus sampling, fixes that by cutting on cumulative probability instead of count. Sort the tokens from most to least likely, add up their probabilities as you go, and keep the smallest set whose total reaches p. That set is the “nucleus”. Everything below it is discarded.
Here is the code, and it is short:
function nucleus(probs, p) {
const ranked = probs
.map((prob, id) => ({ id, prob }))
.sort((a, b) => b.prob - a.prob);
const keep = [];
let cumulative = 0;
for (const token of ranked) {
keep.push(token);
cumulative += token.prob;
if (cumulative >= p) break; // stop the moment we cross the threshold
}
return renormalize(keep); // rescale survivors back to sum 1, then sample
}The adaptivity is the whole point. On a confident step the nucleus is one or two tokens, so top-p behaves almost like greedy. On an uncertain step it opens up to include everything reasonable. That is why top-p is the truncation knob most people reach for, and why many providers expose it right next to temperature.
You rarely need all of these at once. A typical setup is temperature plus one truncation knob. When both are on, they run in order: temperature reshapes the logits first, then top-k or top-p (or min-p) trims the tail, then the sampler draws one token from what survives. A moderate temperature with top-p is a common, sane default: enough variation to feel alive, a cut that keeps it from tumbling into nonsense.
Feel it: a temperature sandbox
Reading about “sharper” and “flatter” only gets you so far. Below is a fixed distribution over five tokens with hardcoded scores, so no model and no network is involved. Drag the temperature and watch the bars breathe. Then hit sample twenty times and watch where the draws actually land: at low temperature almost every draw is the favourite, and as you raise it the tally spreads across the field.
Two things are worth noticing while you play. Slide temperature toward zero and the tally collapses onto blue almost every time, which is greedy in all but name. Slide it up and even neon, the least likely token, starts collecting draws. But neon never overtakes blue, because temperature only reshapes the odds. It does not reorder them or conjure a better answer. That last point is the one people most want to be false.
The determinism trap
Here is the thing everyone gets wrong, and it cost me an afternoon of staring at a “flaky” CI job that was not flaky at all.
Temperature 0, greedy decoding, is the most reproducible mode the API offers. Run the same prompt at temperature 0 and you will very often get the exact same output, again and again. So people reach the natural conclusion: temperature 0 means deterministic, and I can write a test that asserts the model returns this exact string. That test passes locally. It passes in review. It passes two hundred times in CI. Then one morning it fails, the diff shows a single word changed, nothing in your code moved, and you lose the morning.
Temperature 0 is more predictable. It is not a guarantee of byte-identical output. The reason is subtle and worth understanding, because once you get it you will stop writing the broken test.
Greedy decoding picks the token with the highest logit. But the logits themselves are not perfectly stable across runs. Floating-point addition is not associative: add the same numbers in a different order and you can get an answer that differs in the last bit. On a shared inference server your request gets batched together with other people’s requests, and the batch size changes constantly as load rises and falls. A different batch size means the arithmetic inside the model runs in a slightly different order, which nudges the logits by a hair. Most of the time that hair changes nothing. But when the top two tokens are nearly tied, a hair is enough to flip which one wins, and from that token on the two sequences drift apart.
None of that is randomness you dialed in. It is numerical noise in the machinery, and there is more of it: which GPU your request lands on, mixture-of-experts routing, and quiet backend updates all move the logits too. Recent work in 2025 traced most of this instability to a lack of “batch invariance” in the core kernels and showed it is fixable, with batch-invariant implementations now appearing in some open inference engines. Good to know. But the hosted endpoint you call over HTTP makes no such promise by default, so you engineer as if it does not.
So the extraction test from the top of this article should never have looked like the first line here:
// Flaky: assumes byte-identical output. Passes, passes, passes, then fails.
expect(reply).toBe('{"currency":"USD"}');
// Sturdy: parse it and assert the rules you actually care about.
const out = JSON.parse(reply);
expect(out.currency).toMatch(/^[A-Z]{3}$/);
expect(ALLOWED.has(out.currency)).toBe(true);The second version keeps passing whether the model says USD, usd, or wraps it in extra whitespace, and it fails only when the currency is genuinely wrong. That is the discipline of testing applied to a non-deterministic component, and it is the same reason structured output with a schema and a validator beats hoping for a fixed string.
What about seeds?
Some providers let you pass a seed to make sampling reproducible, sometimes alongside a fingerprint of the backend build. Same seed, same input, same fingerprint, and you get the same output most of the time. It is genuinely useful for debugging: pin a seed, reproduce a weird generation, poke at it.
But read that “most of the time” carefully. A seed pins the random draw; it does not pin the logits, so all the numerical noise above still applies. And the moment the provider ships a backend or model update, the fingerprint changes and the reproducibility lapses without warning. Availability varies too: as of 2026 some providers expose a seed, others do not, and the semantics differ between them. Treat a seed as a debugging aid, never as a contract your system depends on.
Reasoning models change the picture
One more wrinkle, because it surprises people. Some models that do extended internal reasoning before answering restrict or ignore the temperature knob entirely. The intuition: on these models the deliberation is what drives accuracy, and the sampling temperature mostly affects surface phrasing, not whether the answer is right. A provider might lock temperature to a fixed value for such a model, or accept the parameter and quietly clamp it. Do not assume the dial is live just because the API accepts the field. Check the specific model’s docs, and confirm with your own outputs.
Picking a setting
Strip away the theory and the practical guidance is short. Match the temperature to how much variety the task actually wants.
The one trap to name out loud: cranking temperature does not make the model smarter. When people get a weak answer at temperature 0 and slide the dial up hoping for a better one, what they get is a weak answer that is now also unpredictable. If the output is wrong at low temperature, it is wrong in more directions at high temperature. The fix for a wrong answer is a better prompt, better context, or a better model, never a hotter sampler.
How this touches evals
Sampling settings and evals interact in ways that will bite you if you are not deliberate.
Run your evals at the temperature you actually ship at. A suite that scores your extractor at temperature 0 tells you almost nothing about the assistant you run at 0.8 in production, because the behaviour you are measuring is not the behaviour your users get. Measure the thing you actually run.
And when an eval starts flaking, ask whether the flakiness is a real regression or just the sampling you dialed in. At higher temperatures the same input produces a spread of outcomes, so a single run is a single roll of the dice. For anything you actually care about, run each case several times and look at the distribution of results (how often it passes, not whether one run passed). A metric that swings twenty points between runs is telling you about variance, not quality, and lowering temperature to make the number stop moving is cheating if the product needs the variety. Measure honestly, then decide.
Summary
- The model produces a probability distribution over next tokens. Sampling is a separate step you control that turns that distribution into one actual token.
- Greedy decoding always takes the top token. It is the most predictable sampler and the right default for anything with a single correct answer.
- Temperature reshapes the distribution before sampling by dividing the logits by
T. LowTsharpens toward the favourite (focused, repetitive); highTflattens it (diverse, surprising); very highTcollapses into incoherence. - Temperature ranges and defaults differ between providers, so a value is not portable. Re-tune when you switch models.
- Top-k keeps a fixed number of top tokens; top-p (nucleus) keeps the smallest set whose cumulative probability reaches a threshold, so it adapts to the model’s confidence. Min-p, anchored to the top token, is a newer option that holds up better at high temperature.
- Temperature 0 is more reproducible but not byte-identical. Floating-point order, request batching, hardware, routing, and backend updates all move the logits, so nearly-tied tokens can flip between runs. Never assume identical output.
- Seeds help reproducibility where offered, tied to a backend fingerprint that changes on updates. Treat them as a debugging aid, not a contract.
- Use near-zero for structured output and anything you parse, moderate for chat, higher for ideation. Higher temperature buys variety, not intelligence, and if you fake a passing eval by lowering it, you have only hidden the problem.