Tokenization — what a model actually reads

Why the pieces exist at all

A model works with a fixed vocabulary of numbers. Text has to be turned into that vocabulary before anything else happens, and the method used by current models is byte pair encoding: start from raw bytes, then repeatedly merge the pairs that occur most often in the training data.

The result is a dictionary of a few hundred thousand pieces where frequency decides the size. the is one token — number 3086 in the o200k_base dictionary. The same word with a space in front of it is also one token, but a different one: number 290. Spaces are usually glued to the start of the word that follows, so where a word sits in a sentence changes which piece it becomes.

Nothing in this process knows about letters, syllables or grammar. It only knows what tends to appear next to what — useful pieces emerge because they are frequent, not because they are meaningful. OpenAI’s own description of the method puts the goal in one phrase:

attempts to let the model see common subwords

What this explains

Once you can see the pieces, several behaviours stop being mysterious.

Letters are invisible. A word that arrives as a single token cannot be spelled or counted from the inside. That is the whole of why AI can’t count the letters in a word.

Numbers are chunked, not read. 1234567890 becomes four tokens — 123, 456, 789, 0. Digits are grouped by frequency, not by place value, so arithmetic on long numbers is working against the input format.

Other languages cost more. English dominates the training data, so English gets the efficient pieces. The German Straßenbahnhaltestelle takes seven tokens where an English word of similar length takes two or three. The same document, translated, is a different size in the only unit that matters for limits and price.

Limits are counted in tokens. Context windows and output limits are measured in these pieces, not in words or characters. If you want to know whether something fits, you have to count the same way the model does — that is what Context fit does with a whole document, and what Token counter does with a piece of text.

Technical detail: which dictionary belongs to which model

OpenAI ships the mapping from model name to encoding in the tiktoken repository, as a plain Python dictionary. GPT-4o and GPT-5 map to o200k_base; the GPT-4 and GPT-3.5 generation maps to cl100k_base. The list stops there — the models OpenAI has released since GPT-5 have no entry — and the file carries its own warning that the map is provisional, so it is worth re-reading rather than memorising.

Anthropic and Google publish no equivalent at all. Any token number quoted for a model with no published encoding — theirs, and OpenAI's newest — including ours, is a character-based estimate, and we mark it as an estimate wherever it appears rather than presenting it as a measurement.