Tokens, and Why They Cost You
A support bot I helped ship looked great in the demo. English question in, English answer out, a couple of cents per conversation. Then we switched it on for users in India, and the same feature started costing roughly three times as much and running out of room halfway through chats that would have been fine in English. Nobody had touched the code. The prompts were the same length on screen.
What changed was something you cannot see in the source. Before the model reads a single thing you send it, your text gets chopped into tokens, and Hindi pays a token tax that English does not. Until you understand tokens, that surprise is invisible. You cannot predict it, price it, or fix it.
Here is the fact the rest of this article hangs on. The model does not read characters, and it does not read words. It reads tokens, and every operational number you care about (your bill, the size of your context window, your rate limit) is counted in tokens, not words. Spend an afternoon on this now and a lot of later confusion just evaporates.
The model sees a list of integers
In what a language model actually does the model was a pure function from a sequence of tokens to a probability for the next one. Time to open up that word “token.”
A tokenizer sits between your string and the model. It takes text and returns a list of integers, one per token, where each integer is an index into a fixed vocabulary the model was built with. That vocabulary is a frozen list of text chunks, usually somewhere between 50,000 and a couple of hundred thousand entries as of 2026. Encoding is the lookup from text to those integer ids. Decoding is the reverse, turning the model’s output ids back into text you can read.
Four words on screen. Six tokens to the model. A few things in that picture surprise everyone the first time.
The space is not its own token. It gets glued to the front of the next word, so ate at the start of a string and ate in the middle are two different tokens with two different ids. That sounds like a nitpick until you are debugging why two prompts that look identical tokenize differently.
cheeseburger is one word but two tokens (cheese and burger), because those two pieces were common enough in the training text to earn their own vocabulary entries while the full compound was not. And the model never sees the letters c-h-e-e-s-e. It sees the single integer that stands for the chunk cheese. The spelling is gone by the time the model looks.
// A tokenizer is just two lookups against a fixed vocabulary.
const ids = encode("I ate a cheeseburger."); // [40, 8391, 261, 33172, 41876, 13]
const text = decode(ids); // "I ate a cheeseburger."
ids.length; // 6 <- this is the number you pay forThose specific ids are illustrative, but the shape is real: encode gives you integers, .length is your token count, and decode is how streamed ids turn back into letters on a screen.
What a token actually is
A token is a chunk of text, usually a subword. The pieces come in three rough sizes and it helps to hold all three in your head:
- A common word is a single token.
the,error,function,returnall map to one id each. Most everyday English text lands here, which is where the friendly rule of thumb comes from. - A rare, long, or made-up word shatters into several.
cheeseburgeris two. Something likeantidisestablishmentarianismor a random identifier likex7bQcan be four, five, or more, because no single chunk of it was frequent enough to deserve one token. - Whitespace and punctuation are tokens too. A newline, a run of spaces, a tab, a Markdown bullet: all cost. Pretty-printed JSON with two-space indentation is meaningfully more tokens than the same JSON minified, for zero benefit to the model.
The famous rule of thumb for English prose is about 4 characters per token, or roughly 0.75 words per token, so 1,000 tokens is around 750 English words. It is genuinely useful for a napkin estimate. It is also only true for ordinary English, and it quietly lies the moment you feed in code, numbers, JSON, or another language. Use it to guess. Never use it to budget.
The vocabulary was learned, by merging
Where does that fixed vocabulary come from? Nobody sits down and writes out 100,000 word-pieces by hand. It is learned from data, most commonly with an algorithm called byte-pair encoding, or BPE. The idea is almost embarrassingly simple.
Start with the smallest possible pieces: individual characters (in modern tokenizers, individual bytes). Then look across a huge pile of training text and find the single most frequent adjacent pair. Merge that pair into a new token and add it to the vocabulary. Look again, find the new most frequent pair, merge it, add it. Repeat tens of thousands of times. Frequent sequences like th, then the, then the, then whole common words, get promoted into single tokens. Rare sequences never win a merge, so they stay in small pieces forever.
That is the whole trick, and it explains the behaviour you saw above. cheese and burger each survived as tokens because they show up constantly. A weird identifier survives only as tiny fragments because it never showed up at all. One more detail matters: modern tokenizers do this over raw bytes, not characters, starting from the 256 possible byte values. That means any text at all (any language, any symbol, any emoji) can always be encoded without an “unknown” token. It just might take a lot of tokens to do it, which is exactly the problem in the next section.
Run the merge loop yourself. This is a tiny BPE over a fixed four-word sample. Hit merge and watch the most frequent adjacent pair fuse into a new token, the vocabulary grow, and the words re-tokenize with fewer, bigger pieces each round.
The merges are chosen once, during the tokenizer’s training, and then frozen. You do not get to retrain them per request. Whatever vocabulary the model shipped with is the one your text gets squeezed through, which brings us to the uncomfortable part.
Some languages cost two or three times more
The merges are picked by frequency in the training text, and that text is overwhelmingly English. So English got the lion’s share of efficient, whole-word tokens. Common English words are one token each. Common Hindi or Japanese words often are not, because their byte sequences never became frequent enough, during that English-heavy training, to earn a merge. They fall back to short fragments, sometimes down to individual bytes. Same meaning, many more tokens.
Real measurements vary by language and by tokenizer, but the pattern is consistent. Many languages run 1.5 to 2 times the English token count for the same content. Hindi, Arabic, and Burmese commonly land around 3 to 4 times. A few scripts can hit close to 10 times in the worst cases. Emoji are their own tax: a plain emoji is often several tokens, and the composed ones (a family, a skin-tone variant, anything built from a zero-width-joiner sequence) can be five or ten tokens for the single glyph you see.
Two consequences fall straight out of that, and both are easy to miss if you only ever test in English.
Cost scales with tokens, so a feature that costs a cent per call in English quietly costs three cents in Hindi. Multiply by real traffic and it is a budget line, not a rounding error. And the context window is measured in tokens too, so a document that fits comfortably in English overflows in Japanese or Telugu. Your users in those languages hit the ceiling sooner, paste in less, and get fewer conversational turns before things start falling out the back. It is a real fairness and reliability gap hiding inside a detail most teams never look at.
Numbers, and why the math is shaky
You saw in what a language model does that models are unreliable at arithmetic. Tokenization is a big part of why. Numbers get chopped in ways that ignore place value, differently across model families, and the boundaries even move when a number gets longer.
Sit with the bottom row, because it is the killer detail. A model that groups digits by three sees 31337 as the two symbols 31 and 337. Add a single digit and 313370 becomes 313 and 370. The 337 unit it had just learned about is gone, replaced by chunks that sit in different columns. There is no stable notion of “the ones place” that survives the number getting longer, so column arithmetic, the thing you learned in school precisely because it is reliable, has no clean footing.
Small sums the model has effectively memorised from seeing them constantly. Large or unusual ones it pattern-matches into something that looks right and often is not. The fix is not a cleverer prompt. It is to stop asking the model to be a calculator and hand real arithmetic to real code, through tool calling or a sandbox. The model is great at deciding a calculation is needed. It is bad at being the calculator.
Where tokens become money and limits
Everything so far has been mechanism. Here is where it touches your job. Tokens are the unit for three separate things, and each one can bite.
Your bill. Providers charge per token, and they meter input and output separately. Output is usually several times more expensive per token than input, because every output token is a full forward pass through the model while your whole input is processed in one shot. As of 2026 the actual rates span a huge range, from fractions of a dollar to tens of dollars per million tokens, and they change month to month, so pin down the current numbers rather than trusting anything you read. The mechanics of pricing and how to keep the bill down get their own treatment in cost and tokens.
Your context window. The model can only attend to a fixed number of tokens at once, and input plus output has to fit under that ceiling. Blow past it and you get an error, or the provider silently drops the oldest content. This is its own topic, the context window, because managing it well is half of building anything that holds a conversation.
Your rate limit. Most providers throttle you on tokens per minute, frequently split into separate input and output budgets, not just requests per minute. That surprises people. One fat 100,000-token prompt can eat your minute as fast as a hundred small ones, so a single user pasting a giant document can throttle everyone else on your key.
Three habits fall out of this, and they are the cheapest wins in the whole part:
- Trim the prompt. Every token of boilerplate, restated instructions, or dumped-in context is paid for on every single call, and again on every retry, because a retry re-sends the entire thing. Long system prompts are convenient and quietly expensive.
- Prefer compact formats. Verbose, pretty-printed JSON is pure overhead to a model that does not care about your indentation. Minify what you send, drop keys it does not need, and never paste a 200-row table when 5 rows answer the question.
- Remember that streaming is token-by-token because generation is. When you watch a reply type itself out (usually delivered over server-sent events), you are seeing the model emit one token, the server flush it, then the next. The chunks arriving are literally tokens, which is why streamed text sometimes breaks mid-word.
Count, don’t guess
The 4-characters rule is fine for a back-of-envelope “will this roughly fit.” For anything that touches money or a hard limit, count with the real tokenizer instead of multiplying. And here is the catch that trips people up: every model family ships its own tokenizer, so a count from one is only an estimate for another. The same prompt might be 1,000 tokens on one model and 1,150 on the next, because their vocabularies were trained separately. Providers hand you the tokenizer for exactly this reason, either as a library you run locally or a count-tokens endpoint you call before you send.
Here is a crude counter to build intuition. Type anything (throw in a long word, a big number, an emoji, some indentation) and watch the estimate move. It splits text with a rough, hand-rolled subword scheme, not a real vocabulary, so treat the number as a ballpark.
Notice the honest habits it nudges you toward. Big numbers fan out into chunks. A long word breaks up. Every space becomes visible because every space is real. The point is not the exact figure (this toy is not any model’s tokenizer) but the reflex: when a count matters, run the real thing and look, rather than eyeballing the word count and hoping.
None of this is glamorous, and that is the point. Tokens are the plumbing under every AI feature you will build. Get comfortable with how text becomes them, why the count is not the word count, and where that count turns into dollars and limits, and the two topics it feeds directly, the context window and cost and tokens, stop feeling like surprises and start feeling like knobs you control.
Summary
- The model never sees characters or words. A tokenizer turns your text into a list of integer ids from a fixed vocabulary (roughly 50,000 to 200,000 entries as of 2026), and those ids are all the model ever reads.
- A token is usually a subword. Common words are one token, a leading space rides with the following word, and rare or long words shatter into several pieces. Whitespace and formatting cost tokens too.
- The vocabulary is learned with byte-pair encoding: start from bytes, repeatedly merge the most frequent adjacent pair, freeze the result. Frequent sequences become single tokens; rare ones never do.
- The rule of thumb is about 4 characters or 0.75 words per token for English prose. It is fine for guessing and wrong for code, numbers, and other languages. Measure when it matters.
- Because training text is mostly English, other languages cost more tokens for the same meaning, often 2 to 3 times, sometimes far more. That means higher cost and less context room for those users.
- Numbers tokenize inconsistently, by groups of three or per digit depending on the model, and boundaries shift as a number grows. That instability is a big reason arithmetic is unreliable, so hand real math to real code.
- Tokens are the unit for your bill (input and output metered separately, output usually pricier), your context window, and your rate limit (often tokens per minute). Trimming prompts and using compact formats are the cheapest wins you have.