Cost, Tokens and Rate Limits
A document-Q&A feature I helped ship cost about a cent a question in the demo. We turned it on for real users. Within a week the AI line on the invoice had four figures on it, and traffic had barely moved.
The demo asked one question against one document. Real users asked twelve follow-ups against the same document, and every follow-up re-sent the whole document plus the whole conversation so far as fresh input. The model keeps nothing between calls. Each turn you pay for everything in the window again, from scratch.
That is the thing nobody warns you about. Unlike almost every other kind of software, an AI feature has a genuine marginal cost per request, and it grows with usage. A page of static HTML costs the same to serve whether one person loads it or a million do. A model call costs money every single time, and the amount depends on how much text goes in and how much comes out. So cost is not something you tune later once you have “scale problems.” It is an engineering constraint from the first line you write.
Where the money goes
Every model call is billed on two numbers: how many tokens go in, and how many come out. Input and output are metered separately, priced separately, and output is the expensive one.
Input is everything you send: the system prompt, the conversation history, any retrieved context, and the user’s actual message. All of it counts, and all of it counts again on every turn, because the API is stateless. There is no session on the model’s side quietly remembering turn 1. The context window is the budget those input tokens fill, and you refill it every call.
Output is what the model generates, one token at a time.
Why is output pricier? It is mechanical, not arbitrary. Your entire input is read in a single forward pass through the model. But every output token is its own forward pass, generated one at a time, each one conditioned on all the tokens before it. Reading is cheap. Writing is expensive. As of 2026, output commonly runs somewhere around four to five times the input rate, sometimes more, and that ratio has held steady across providers even as the absolute prices fall.
One consequence trips people up constantly: a short prompt with a long answer can cost more than a long prompt with a short answer. Cost does not track “how much did I type.” It tracks tokens, and output tokens carry more weight each.
// Every response comes back with a usage report. Log it on every call.
const { usage } = response;
usage.input_tokens; // full-price input
usage.cached_input_tokens; // served from cache, billed at a fraction (below)
usage.output_tokens; // the pricey half
// Field names vary by provider, but the three buckets are always there.
const cost =
(usage.input_tokens / 1e6) * INPUT_PER_M +
(usage.cached_input_tokens / 1e6) * INPUT_PER_M * 0.1 +
(usage.output_tokens / 1e6) * OUTPUT_PER_M;If you take one habit from this article, take that one. Read the token usage off every response and record it. You cannot control a number you never look at.
Output is the half you cannot see coming
Because output is priced high and the model decides how long it is, output is where the surprises live. Three defences:
Cap it. Set a maximum output length on every call. It is your only hard ceiling on the expensive half. A model that decides to “explain its reasoning” at length, or a prompt that accidentally asks for the whole list instead of the top five, hits the cap instead of your budget.
Watch reasoning models. A reasoning model spends output tokens thinking before it answers, and you pay for every one of those thinking tokens even though the user never sees them. On an easy task, a reasoning model can cost several times a plain one for the same visible reply. Use the thinking budget deliberately; do not leave it cranked up on requests that do not need it.
Ask for less. “Answer in one sentence” is a cost instruction as much as a style one. So is “return only the JSON, no explanation.”
The stable prefix is almost free: prompt caching
This is the biggest lever most teams never pull.
Providers can cache a stable prefix of your prompt. The first call that establishes a prefix pays full price, plus a small write premium. Every later call that begins with the exact same prefix is billed for that portion at a fraction of the input rate, commonly around a tenth. You pay full price only for the part that actually changed.
Two mechanics decide whether this works for you.
It is a prefix match. The cache keys on the exact bytes from the start of your prompt up to a breakpoint. One character different anywhere in that prefix and the match is gone, and you pay full price for the whole thing again. So the discipline is simple: put the stable stuff first (a fixed system prompt, fixed instructions, fixed context), keep it byte-for-byte identical across calls, and put the volatile stuff (the user’s question) last.
And the minimum cacheable prefix is not tiny, usually on the order of one to a few thousand tokens depending on the model. A short prompt silently will not cache even with the flag set. Check the token usage: if the cached-tokens count stays at zero across repeated calls that should be sharing a prefix, something up front is varying.
That RAG example from the diagram is real, not hypothetical. An 8,000-token context re-sent on every question costs roughly $24 per million questions at a $3-per-million input rate. Cached, the same context costs around $0.30 per million. Same feature, same answers, a fraction of the cost, from putting the fixed part first and leaving it alone.
Right-size the model
Most requests are easy. A small, cheap, fast model handles them perfectly well. The big expensive model earns its keep only on the slice that genuinely needs it. Sending everything to the top model is, most of the time, just overpaying.
The arithmetic is stark. Say a million requests a month, each around 2,000 input and 300 output tokens. Send all of it to a top-tier model at roughly $10 in and $50 out per million, and you are looking at something like $35,000. Route the easy 90% to a cheap tier at $1 in and $5 out, and keep the hard 10% on the big model, and the same month lands near $5,000. No quality loss on the calls that were always going to be easy.
How you decide which requests are easy (a cheap classifier, a rules check, or try-cheap-then-escalate-on-a-bad-answer) is its own topic, covered in model routing and fallbacks. For the budget, the takeaway is blunt: one model for everything is almost always the wrong default.
Send less, and do not ask twice
The rest of the levers are smaller, and mostly point back to topics with their own chapters. Quick pass:
- Trim the input. Every token of restated instructions, dumped-in context, or the 200-row table when five rows answered the question is paid on every call, and again on every retry, because a retry re-sends the entire thing. Deciding what earns a place in the window is context engineering, and it is a cost discipline as much as a quality one.
- Prefer compact formats. Pretty-printed JSON with two-space indentation is pure overhead to a model that does not care about your whitespace. Minify what you send and drop keys the model does not need.
- Cache whole responses for repeated queries so you skip the call, and the bill, entirely (caching model responses).
The other wall: rate limits
Cost is one resource wall. Rate limits are the other, and they bite at scale even when you can happily afford the tokens.
Providers cap you on two axes, usually per minute: requests per minute (RPM) and tokens per minute (TPM), often split into separate input and output budgets, sometimes with a daily token cap on top. TPM is usually the one you hit first. Higher tiers, earned by spending more over time, get higher ceilings.
Three things surprise people here.
One fat request can eat the whole minute. A single user pasting a 100,000-token document blows the TPM budget for everyone sharing that API key. Per-user quotas, a limit you enforce on your side before the request ever goes out, stop one user starving the rest. This is ordinary rate limiting applied to a dependency you do not own.
A 429 is a “wait,” not a “fail.” When you cross a limit you get a 429 Too Many Requests, usually with a Retry-After header telling you exactly how many seconds to wait. Honor it. Then retry with backoff and jitter so a burst of retries does not all stampede back the instant the window resets. Most SDKs retry 429s for you with sane defaults; know whether yours does before you build your own.
Batch the work that is not urgent. For jobs that do not need an answer this second (nightly summaries, backfilling embeddings, running evals over a big set) most providers offer a batch endpoint at roughly half price, in exchange for “results within a few hours instead of now.” If you do not need it immediately, that is a 50% discount for free.
Know the number before you ship
The estimate is arithmetic, and you should do it before you write the feature, not after the invoice arrives. Tokens per request, times the price per token, times the number of requests. Do input and output separately, because the rates differ.
// Back-of-envelope, before you write the feature.
const INPUT_PER_M = 3; // $ per 1M input tokens (illustrative, changes)
const OUTPUT_PER_M = 15; // $ per 1M output tokens (illustrative, changes)
const inputTokens = 6_900; // system + history + context + message
const outputTokens = 500;
const requestsPerDay = 20_000;
const perRequest =
(inputTokens / 1e6) * INPUT_PER_M +
(outputTokens / 1e6) * OUTPUT_PER_M;
const monthly = perRequest * requestsPerDay * 30;
// perRequest is about $0.028, monthly about $17,000Seventeen thousand dollars a month is a number you want to meet on a whiteboard, not on a statement. Run it early, and half the “surprising” bills stop being surprising.
Once the feature is live, attribute cost per feature and per user, or you are flying blind. When the bill jumps, you want to answer “which feature, which users” in minutes, and that means tracing every call with its token counts and cost attached. A feature you cannot measure is a feature you cannot optimise.
Here is the whole thing to play with. Drag the sliders and watch the monthly figure move. Notice how much output length and model choice swing it, and how far the cache-hit slider pulls the input cost down once your prefix is stable.
Play with it for a minute and the shape of the whole problem shows up. Crank output length and the total climbs fast, because output is the pricey half. Flip to the premium model and it jumps roughly tenfold. Push the cache-hit slider up and the input cost melts while output does not move at all, because output never caches. Those are the levers, all in one place, and now they are knobs you turn on purpose rather than numbers that turn up on a statement.
Summary
- An AI feature has a real marginal cost per request that scales with usage, unlike static or cached software. Treat cost as an engineering constraint from day one, not a later problem.
- You pay per input token and per output token, metered and priced separately. Output usually costs several times more per token, because every output token is a full forward pass while your whole input is read in one shot.
- Everything in the window is input you pay for on every turn: system prompt, history, and retrieved context, re-sent each call because the API is stateless. Read the token usage off every response and log it.
- Cap the output length on every call. It is your only hard ceiling on the expensive half, and reasoning models bill you for hidden thinking tokens too.
- Prompt caching is the biggest missed lever: a stable prefix is billed at roughly a tenth on repeat calls. Put stable content first, keep it byte-identical, and a stray timestamp or UUID up front silently disables it.
- Route by difficulty. A cheap model for the easy majority and an expensive one for the hard slice can cut the model bill by close to an order of magnitude, since tiers often differ by 10x.
- Trim context, prefer compact formats, and cache whole responses for repeated queries to skip calls entirely.
- Rate limits are the other wall: requests and tokens per minute, with TPM usually binding. Handle 429s with backoff and jitter, honor
Retry-After, add per-user quotas so one user cannot starve the key, and batch non-urgent work at roughly half price. - Estimate before you ship (tokens times price times volume), attribute cost per feature and per user, and put hard spend caps plus alerts underneath so a bug or an abuser hits a wall instead of your wallet.