Reasoning Models and Thinking Budgets
The refund checker was the boring kind of feature: given an order and our returns policy, decide full refund, partial refund, or nothing, and show the number. A standard model handled the common cases fine. “Unopened, bought three days ago, inside the window” is not a hard question. Then it started getting the gnarly ones confidently wrong. A plan changed mid-cycle. A partial refund prorated across two price tiers. A store-credit rule that only kicked in above a threshold. The model would hand back a clean, plausible, wrong number, the kind of case where a human reaches for a scratchpad.
So I swapped the endpoint to a reasoning model. Correctness on the hard cases jumped. Two other things jumped with it. The p95 latency went from about two seconds to the high thirties, and the monthly bill roughly six-x’d. I was now paying for thousands of hidden thinking tokens on every single call, including the eighty percent that were “unopened, three days, full refund, done” and never needed a scratchpad at all.
The fix was not the model. It was deciding, per request, whether this was a two-second question or a thirty-second one, and only paying the reasoning tax when the question earned it. That decision is the actual skill, and it is what this article is about.
What the extra phase is
A standard model answers in one pass. Tokens go in, and it starts emitting the answer immediately, one token at a time (that mechanism is how a language model works). There is no separate “thinking” step. The catch is that a one-pass model has to commit to the first token of its answer before it has worked anything out. For “the sky is…” that is fine. For “prorate this refund across a mid-cycle tier change” it is a setup for a confident guess.
A reasoning model adds a phase in front. Before it writes the answer you see, it generates a private stretch of tokens where it works the problem out: sketches a plan, tries an intermediate step, notices a mistake, backs up, tries again. Then it writes the final answer. Those private tokens are usually called reasoning tokens or thinking tokens. Mechanically they are ordinary tokens, generated left to right like everything else. The difference is that the model was trained (with reinforcement learning) to use that space to deliberate, and the provider usually keeps it out of the reply you get back.
This measurably helps on a specific shape of problem: maths, logic puzzles, multi-step planning, hard code, anything where getting it right means holding several steps in your head and checking your own work. On those tasks a good reasoning model is meaningfully more accurate than the same-generation standard model. On a plain “summarise this email” it is not more accurate. It is just slower and pricier for no gain. The whole game is knowing which task you have.
It is not free, and the meter is hidden
Here is the part that surprises people the first time they read a bill. Those reasoning tokens are generated, and they are billed, even though you usually never see them. Most providers price them as output tokens, at the output rate.
A response object makes it concrete. You asked one question, got back a short answer, and the usage looks like this:
{
"usage": {
"input_tokens": 320,
"output_tokens": 8500,
"output_tokens_details": {
"reasoning_tokens": 8000
}
}
}You can read maybe 500 tokens of answer. You are paying for 8,500 output tokens, because 8,000 of them were thinking you never saw. That is not a rounding error. On a hard prompt at high effort the reasoning can dwarf the answer several times over, and it is the same tokens-are-money accounting from tokens and why they cost you, just with most of the tokens invisible.
The second cost is time. A reasoning model is commonly a few times slower than a standard one, and on genuinely hard problems at high effort the first visible token can be tens of seconds out, because the model is busy thinking and has not started the answer yet. Slower and dearer is the deal you are signing. Sometimes it is a great deal. Often it is a terrible one.
The dial: thinking budgets and effort
Because the tax scales with how much the model thinks, most providers give you a knob to set how hard it deliberates. It shows up in two shapes, and as of 2026 the exact names shift between providers and change often, so treat these as the pattern rather than an API you can copy.
One shape is an effort level, usually low, medium, and high (some providers add a none or minimal at the bottom and an extra-high tier at the top). Higher effort means the model is allowed to think longer, spends more tokens, and takes more wall-clock time. The other shape is a token budget: you hand it a number that caps how many thinking tokens it may spend before it has to start answering.
// Two shapes you will meet. Names differ by provider and change often, so
// check current docs; the idea is stable even when the field names are not.
// (1) an effort level
const reply = await model.generate({
input: messages,
reasoning: { effort: "high" }, // "low" | "medium" | "high", sometimes more
});
// (2) a hard cap on thinking tokens
const reply2 = await model.generate({
input: messages,
thinking: { type: "enabled", budgetTokens: 4000 }, // a ceiling, not a target
});The word “budget” is doing honest work there. It is a ceiling, not a target. A decent reasoning model spends adaptively under the cap: a trivial prompt might use a few hundred thinking tokens even with the budget set to four thousand, while a hard one spends right up to the limit. So raising the budget does not force more thinking on easy prompts, it just raises the roof for the ones that need it.
Where “think step by step” came from
If this feels familiar, it should. Before any of this was baked into models, people got a plain model to reason by asking it to. You appended something like “let’s think step by step, show your working, then give the final answer” to the prompt, and accuracy on maths and logic went up, sometimes a lot. This is chain-of-thought prompting, and it works for a simple reason: by forcing the model to spend visible tokens laying out intermediate steps, you stop it from having to blurt the answer in the first token. It gets room to work.
Solve this problem. Think step by step and show your working,
then put the final answer on the last line by itself.A reasoning model is, more or less, that trick trained into the weights and moved into a dedicated space. The deliberation happens whether or not you ask for it, the model is much better at it than a hand-rolled prompt, and the working usually lives in the hidden reasoning channel instead of cluttering your output. Chain-of-thought prompting still has its uses: it is the poor man’s reasoning model, it works on cheap standard models, and it is handy when you actually want the steps visible in the reply. But for hard problems, a purpose-built reasoning model beats bolting “think step by step” onto a model that was not trained for it.
Do not reason on everything
Back to the bill from the top of this page. The mistake was not using a reasoning model. It was using one on every request. The engineering move is to send the bulk of your traffic to a fast standard model, and escalate to a reasoning model only for the step that genuinely needs it.
Most calls in a real product are not hard. Pulling three fields out of a document, classifying a ticket into one of five buckets, formatting a reply, routing a message, holding a normal chat turn: none of that benefits from a thirty-second think, and paying for one makes your product slower and more expensive for no accuracy in return. Reserve the reasoning model for the actual hard step, and pay the tax only there.
The router itself can be dumb. Sometimes it is a keyword or a heuristic, sometimes a cheap fast-model call that just answers “does this need careful reasoning, yes or no”. Either way it costs a fraction of a reasoning call, and it keeps the expensive model off the easy path.
// Cheap first pass decides whether this call even needs the expensive model.
async function answer(question) {
const kind = await classify(question); // fast, cheap standard-model call
if (kind === "hard") {
return generate(question, {
model: REASONING,
reasoning: { effort: "high" },
});
}
return generate(question, { model: FAST }); // the common path, no reasoning tax
}This is one instance of a bigger pattern, picking the cheapest model that can do the job and only climbing when you must, which gets its own treatment in fallbacks and routing. For now the rule of thumb is enough: reasoning model for the hard step, fast model for everything else.
See it think: fast versus reasoning
Below is a tiny simulated solver. It is not a real model and it makes no network calls. It has a handful of classic trick questions where the fast, intuitive answer is wrong, plus the step-by-step working that gets each one right. In Fast mode it jumps straight to a guess and is usually wrong, the way a one-pass model grabs the first plausible answer. In Reasoning mode it streams its working one step at a time, takes longer, and lands on the right answer, as long as you give it enough budget. Drop the budget too low and watch it run out of room and guess anyway.
Two things to notice while you play. First, fast mode fails on exactly the questions where the obvious answer is a trap, which is the real failure mode of one-pass answers on hard prompts. Second, the reasoning mode is only reliable when the budget covers the work. Cut it short and you get a confidently wrong guess after paying for partial thinking, which is the worst of both worlds and a real thing that happens when you set the budget too low for the task.
Streaming: the user waits through the thinking
With a standard model you stream the response into the UI and the first token shows up fast, so the interface feels alive almost immediately. A reasoning model breaks that. The model is thinking first, so the first token of the actual answer can be many seconds away, and if you just wait for it the user stares at a dead screen the whole time.
So plan the wait. At minimum, show an explicit “thinking” state so the user knows work is happening rather than assuming the app hung. Better, if the provider exposes it, stream a running summary of the reasoning so there is motion on screen while the model works.
That “if the provider exposes it” is important, because most providers do not hand you the raw chain of thought. You typically get one of three things: nothing until the answer, a short human-readable summary of the reasoning, or the answer plus that summary. The raw thinking is usually withheld on purpose, partly competitive and partly for safety, since the unfiltered reasoning trace can contain things they would rather not surface. So build your UI around a summary or a status indicator, not around the assumption that you can show the model’s actual private working.
// Provider event names vary; the shape is the pattern. Some emit a reasoning
// summary you can render as a "thinking" indicator, then the real answer.
for await (const event of stream) {
if (event.type === "reasoning.delta") showThinking(event.text); // a summary, if any
if (event.type === "answer.delta") showAnswer(event.text); // the reply itself
}One more practical note. Because thinking can run long, your request timeouts and retry logic need headroom. A budget you tuned for two-second standard calls will time out a forty-second reasoning call and turn a slow-but-correct answer into a failed one.
How reasoning models touch the rest of your stack
A few sharp edges once you actually wire one in.
Context and multi-turn. Within a single turn, the thinking tokens count against the model’s context window, so a deep think eats into the budget you have for input and output. Across turns you usually get a break: most providers discard the reasoning tokens between turns by default, so they do not pile up in the conversation history. The exception worth knowing is agent-style tool loops, where some providers hand you back an opaque or encrypted reasoning block that you must pass in with the next call for the loop to work correctly. If you get one, keep it and send it back; do not drop it.
Temperature. Many reasoning models restrict or ignore the temperature knob entirely. The reasoning drives the accuracy, and the sampling temperature mostly moves surface phrasing, so a provider may lock it to a fixed value or accept the parameter and quietly clamp it. Do not assume the dial is live just because the API takes the field.
Evals. When you measure a reasoning model, run the eval at the same effort level you actually ship at, because behaviour and cost both change with effort and a suite scored at high effort tells you little about a service you run at low. And watch the price of the suite itself: every eval case now spends reasoning tokens, so a large suite at high effort gets expensive fast. Most importantly, do not score accuracy in isolation. The entire point of this class of model is a tradeoff, so measure accuracy against cost and latency together, or you will happily ship a model that is two points more accurate and ten times slower for a task that never needed it.
Summary
- A reasoning model adds a private phase before its answer, spending hidden reasoning tokens to plan, try steps, and check its own work. This measurably helps on maths, logic, multi-step planning, and hard code, and does nothing for easy tasks.
- Those reasoning tokens are generated and usually billed as output tokens even though you never see them, and they add latency, often several times a standard call and sometimes tens of seconds to the first token.
- Providers give you a dial: an effort level (commonly low, medium, high) or a token budget that caps the thinking. A budget is a ceiling, not a target; the model spends less on easy prompts. Names and defaults shift often, so check current docs.
- “Let’s think step by step” prompting was the hand-rolled ancestor of this. Reasoning models bake that deliberation into training and move it into a dedicated, usually hidden channel.
- Do not reason on everything. Send the bulk of traffic to a fast standard model and escalate only the genuinely hard step. Reasoning models are overkill for extraction, classification, and formatting, and can overthink an easy task into a wrong answer.
- Streaming does not save you here: the model is silent while it thinks, so show a thinking state or stream a summary. Most providers withhold the raw chain of thought and give you a summary at best.
- Mind the edges: thinking tokens eat the context window within a turn, temperature is often locked, evals must run at your shipped effort and weigh accuracy against cost and latency, and timeouts need headroom.