How text is chopped into the pieces the model can see, and what the chopping does to it.
Every chapter so far started at "token ids in". Chapter 1 promised to explain where the ids come from, and this is that chapter. It sits after the model rather than before it for a reason: the choices in a tokenizer only make sense once you know what they cost downstream, in embedding rows (Chapter 2), in T² attention (Chapter 3), and in what the model can and cannot learn to see (Chapter 5). Chapter 7 then puts the whole pipeline, tokenizer included, into its two loops.
The question this chapter answers: what is a token, how is the vocabulary decided, and why do models behave strangely around spelling, numbers, and languages other than English?
A court stenographer does not type letters. She types chords: one keystroke for "the", one for "-ing", one for "objection". Her machine has a dictionary of a few thousand such chunks, chosen because they come up constantly in courtrooms. Common words are one stroke. A rare surname is spelled out in pieces. The result is a transcript that is much shorter, in keystrokes, than the letters it represents, and that can still spell anything.
The dictionary was not designed by a linguist. It was built by frequency: whatever came up most often got its own chord first. And it was built for courtrooms; hand the same stenographer a chemistry lecture and her strokes-per-word rate doubles, because "trifluoromethyl" is not in the dictionary.
A tokenizer is that dictionary. The chords are tokens. Building it by frequency is byte-pair encoding. And the chemistry lecture is every language, domain, and format that was underrepresented when the dictionary was built.
| In the picture | In the machine | The word we will use |
|---|---|---|
| One chord | A chunk of text with an integer id | token |
| The chord dictionary | The fixed list of all tokens, size V | vocabulary |
| Building the dictionary by frequency | Repeatedly merge the most frequent adjacent pair into a new token | byte-pair encoding (BPE), the merge list |
| Spelling a rare name in pieces | Fall back to smaller tokens, down to single bytes | byte fallback |
| Rules about where a chord may start | Split text on spaces, punctuation, and digit runs before merging | pre-tokenization |
| Reserved strokes for "new speaker", "end of testimony" | Tokens with no text, used to mark structure | special tokens |
| Strokes per word in a chemistry lecture | How many tokens a text costs, by language and domain | tokenization efficiency (tokens per word, bytes per token) |
Two obvious choices for the unit, and both fail for reasons you can now cost out.
| Unit | Vocabulary | A 1,000-word document becomes | What breaks |
|---|---|---|---|
| Characters (or bytes) | ≈ 256 | ≈ 5,500 tokens | Attention is T²: 30 million score entries per head per layer instead of 1.6 million. Every prediction is "the next letter", so the model must spend layers re-assembling words before it can think about them. Context windows shrink 4× in words. |
| Words | unbounded | ≈ 1,000 tokens | Every misspelling, name, URL, and code identifier is a new word. A million-entry table still meets unknown words daily, and an unknown word has no row in W_E: the model literally cannot see it. "Running" and "runner" share nothing. Other languages need their own million. |
| Subword pieces (BPE) | ≈ 32k to 256k | ≈ 1,300 tokens | Common words are one token; rare ones are a few; anything at all can be spelled from bytes. The compromise every frontier model makes public. |
The middle column is the point. Tokens are a compression of the text, and the compression ratio sets how much text fits in a context and how much attention costs. The right column is the other constraint: the vocabulary must be closed (a fixed V for the softmax of Chapter 1) and yet able to represent anything. Subword tokenization is what satisfies both.
The algorithm is short enough to run on paper. Start with every word spelled out in characters, with an end-of-word marker. Count every adjacent pair. Merge the most frequent pair into a new token, everywhere it occurs. Repeat until the vocabulary has the size you want. The merges, in order, are the tokenizer: to tokenize new text, apply the same merges in the same order.
corpus: low lower lowest slow slower newer newest wide wider widest
start: l o w · l o w e r · l o w e s t · s l o w · s l o w e r · n e w e r · n e w e s t · w i d e · w i d e r · w i d e s t ·
("·" marks the end of a word, so merges never cross word boundaries)
pair counts (top few): (l,o) 5 (o,w) 5 (e,r) 4 (r,·) 4 (e,s) 3 (s,t) 3 (w,i) 3 …
merge 1: l + o → "lo" lowest is now lo w e s t ·
merge 2: lo + w → "low" low e s t ·
merge 3: e + r → "er"
merge 4: er + · → "er·" a suffix token: "-er" at the end of a word
merge 5: e + s → "es"
merge 6: es + t → "est"
merge 7: est + · → "est·" another suffix: lowest is now low est·
merge 8: w + i → "wi"
$ python code/ch06/bpe_by_hand.pymerge 1: 'l'+'o' (seen 5×) 'lowest' → lo w e s t merge 2: 'lo'+'w' (seen 5×) 'lowest' → low e s t merge 3: 'e'+'r' (seen 4×) 'lowest' → low e s t merge 4: 'er'+'' (seen 4×) 'lowest' → low e s t merge 5: 'e'+'s' (seen 3×) 'lowest' → low es t merge 6: 'es'+'t' (seen 3×) 'lowest' → low est merge 7: 'est'+'' (seen 3×) 'lowest' → low est merge 8: 'w'+'i' (seen 3×) 'lowest' → low est vocabulary after 8 merges: 10 tokens ['est', 'er', '', 'low', 'wi', 'd', 'e', 'n', 's', 'w']
Nobody told the algorithm that "-er" and "-est" are suffixes. They came out of frequency: "e r ·" occurred at the end of four words, so it was merged early. This is the sense in which a tokenizer "knows" morphology: only as much as the counts happened to encode. "lowest" is two tokens, low + est·, which is a linguistically reasonable split. "wider" became wi + d + er·, which is not. Both are what the counts produced.
Now watch it at a scale where words become single tokens. The widget trains BPE in your browser on a few paragraphs of this book.
Three refinements turn the paper algorithm into what frontier models ship.
Bytes, not characters. Start the vocabulary from the 256 possible byte values rather than from characters. Then any string in any script, any emoji, any binary junk, can be spelled from the base vocabulary, and the merges learned on top only make common sequences cheaper. Nothing is ever "unknown". The cost is that a character outside the training distribution can cost three or four tokens, one per UTF-8 byte.
Pre-tokenization. Before merging, the text is split by a regular expression into chunks that merges may not cross: typically at spaces (with the space attached to the following word, which is why " the" and "the" are different tokens), at punctuation, and, in current tokenizers, into runs of at most three digits public (Llama 3 and GPT-4 both do this). Without the digit rule, "2024" and "2025" would be single, unrelated tokens and arithmetic would be memorisation; with it, every number is built from a small set of 1- to 3-digit pieces the model sees constantly.
Special tokens. A few ids are reserved for structure rather than text: beginning of document, end of document, and, after post-training (Chapter 14), markers for the start and end of a user turn or an assistant turn. They have embedding rows like any token and the model learns what they mean from the loss like any token; the only special thing about them is that the tokenizer never produces them from ordinary text, so user input cannot forge them.
tiktoken library: 100k merges inherited from an earlier OpenAI tokenizer plus 28k added to improve non-English coverage public. Its predecessor, Llama 2, had 32k entries; the change alone cut tokens-per-text by about 15% on English and much more on other languages public.Every famous "LLMs are bad at X" where X is spelling, counting letters, arithmetic, or rhymes has the same root: the model does not see letters. It sees ids. What it knows about the inside of a token, it had to learn from statistics of how that token co-occurs with others.
Counting letters. "strawberry" is typically two or three tokens in frontier tokenizers public. To count its r's the model has to recall the spelling of each piece, and it has seen the spelling of "berry" written out far less often than it has seen the word used. Models have improved at this through training data that spells things out and through reasoning at generation time (Chapter 16), not through seeing letters.
Arithmetic. With the three-digit rule, 50000 is 500 + 00: the split does not align with place value. Carrying across that boundary is a learned skill, not a given. Models are noticeably better at arithmetic on numbers whose tokenization is regular, and tokenizer design has been changed specifically to help public.
Code and whitespace. Early tokenizers had no token for four spaces, so every indented Python line cost four tokens before the code began. Modern vocabularies include runs of spaces and common code fragments, because code is a large fraction of the training mix public. A tokenizer trained on the wrong mix makes an entire domain two to three times more expensive.
Languages. Because merges are learned by frequency and English dominates the training corpus, English is cheap and everything else costs more. Published measurements on 100k-class vocabularies put many non-Latin-script languages at two to four times the tokens per sentence of English public. Every consequence follows: less of a Japanese document fits in the context, each Japanese reply costs more, and the model has had fewer "effective words" of Japanese per byte of training data. Growing the vocabulary with language-specific merges, as Llama 3 did, is the standard mitigation.
Glitch tokens. A vocabulary is built from a sample of text that may not match the training corpus. Tokens that were frequent in the sample but nearly absent from training end up with embedding rows that were never moved from their random initialisation (Chapter 5: a row is updated only when its token appears). Prompting a model with such a token produces bizarre behaviour, the best known being a Reddit username that made early GPT models babble public. Modern pipelines check for these.
| Quantity | Llama 3 | Evidence |
|---|---|---|
Vocabulary V | 128,256 | public |
| Type | byte-level BPE, tiktoken-style pre-tokenization, digit runs ≤ 3 | public |
| Special tokens | 256 reserved, including begin/end of text and chat-turn headers | public |
| Bytes per token on the training mix | ≈ 3.9 (vs 3.2 for Llama 2's 32k vocabulary) | public |
| Closed frontier models | Vocabularies of roughly 100k to 260k; tokenizers are usually inspectable via a counting endpoint even when not published | inferred |
Why not a million-entry vocabulary? Two costs grow with V and one benefit shrinks.
V = 128k, C = 4096: embedding + unembedding = 2 × 128k × 4096 ≈ 1.05 B parameters V = 1M, C = 4096: 2 × 1M × 4096 ≈ 8.2 B parameters (bigger than the rest of an 8B model) unembedding cost per token: 2 × C × V = 2 × 4096 × 1M ≈ 8 G ops (vs ≈16 G for all 32 layers of the 8B model's MLPs) benefit: tokens per word falls from ≈1.25 to ≈1.15, about 8% shorter sequences
And the third cost has no formula: a token that appears a hundred times in 15 trillion tokens gets a hundred gradient updates to its embedding row, which is not enough to learn what it means. A million-entry vocabulary is mostly such tokens. The 100k–260k range is where the curve of sequence length has flattened and the tables are still a small fraction of the model.
Every provider bills per token, and most expose a counting endpoint so you can see the tokenizer's decisions without inferring them. The script runs six strings through it: prose, the strawberry question, Python, a large number, French, Japanese.
# code/ch06/count_tokens.py
import anthropic
client = anthropic.Anthropic()
samples = ["The detective knew the killer was still in the room.",
"how many r's are in strawberry?",
"def push(queue, item):\n queue.append(item)\n return len(queue)",
"1234567890 + 9876543210 = 11111111100",
"Le détective savait que le tueur était encore dans la pièce.",
"探偵は犯人がまだ部屋にいることを知っていた。"]
for text in samples:
n = client.messages.count_tokens(model="claude-opus-5",
messages=[{"role": "user", "content": text}]).input_tokens
print(f"{n:4d} tokens {len(text):4d} chars {len(text)/n:5.2f} chars/token {text[:40]!r}")(example output — run it to see your provider's real counts; the shape is what matters)
19 tokens 52 chars 2.74 chars/token 'The detective knew the killer was still '
15 tokens 31 chars 2.07 chars/token "how many r's are in strawberry?"
26 tokens 63 chars 2.42 chars/token 'def push(queue, item):\n queue.append('
27 tokens 38 chars 1.41 chars/token '1234567890 + 9876543210 = 11111111100'
27 tokens 61 chars 2.26 chars/token 'Le détective savait que le tueur était e'
31 tokens 22 chars 0.71 chars/token '探偵は犯人がまだ部屋にいることを知っていた。'
Note that a request's token count includes a few tokens of structure (the turn markers) beyond the text, so short strings look slightly more expensive than their content. The ratios between rows are the lesson: numbers and non-Latin scripts cost several times more per character than English prose.
Tokenize by whitespace, no subwords. The vocabulary is every distinct word in the corpus, tens of millions of entries, most seen a handful of times and hence with untrained embeddings. Any new word at inference has no id at all. The model is fluent on common words and blind to everything else, including its own users' names.
Drop the byte fallback. The vocabulary is a fixed set of character-level pieces from the training sample. A script or symbol not in the sample cannot be represented; the tokenizer must emit an "unknown" token, and the model cannot distinguish one unknown from another. Multilingual and code robustness collapse.
Drop pre-tokenization. Merges may cross spaces and punctuation. "the·detective" becomes a candidate token; so does "queue·depth·>". The vocabulary fills with phrase fragments that are frequent in the sample and useless elsewhere, and the same word gets a different id depending on its neighbours. Compression improves slightly; generalisation gets much worse.
Let digit runs be arbitrarily long. "50000" is one token, "50001" another, and "500000" a third, with no shared structure. The model must memorise arithmetic facts per token rather than learning place-value rules over a small set of digit pieces. Published ablations show measurable damage to arithmetic public.
Train the tokenizer on English only, then train the model on everything. Every other language is spelled in near-bytes, costing 3 to 10 tokens per word. The model still learns those languages, but with a fraction of the effective context and at several times the cost, and the T² of attention makes every non-English document disproportionately expensive to train on. This is not hypothetical; it describes several early models.
Say it back. The model needs a closed vocabulary for its softmax and an open one for the world, and subword tokens are the compromise: common strings become single tokens, rare ones are spelled from pieces, and a base of 256 byte values guarantees that any string at all can be spelled. The vocabulary is built by byte-pair encoding: split text into chunks that merges may not cross, then repeatedly glue the most frequent adjacent pair into a new token, recording each merge; the ordered merge list is the tokenizer. Frequency, not linguistics, decides what becomes a token, so suffixes and common words emerge and rare words fragment. A handful of reserved ids mark structure. The consequences follow from "the model sees ids, not letters": spelling, letter counting, and arithmetic across digit-group boundaries are learned facts about tokens rather than perceptions; languages and domains underrepresented when the vocabulary was built cost several times more tokens per word forever after; and a token that rarely appears in training keeps an untrained embedding. Vocabulary size trades sequence length against table size and per-token unembedding cost, and the 100k to 260k range is where that trade flattens. Beacon's tokenizer is a 128k byte-level BPE with three-digit number chunks and 256 special tokens.
"aaab aab ab abab" (with end-of-word markers) for four merges. Write the pair counts before each merge and the vocabulary after. Then tokenize the unseen word "aaabab" with your merge list.C = 4096?bpe_by_hand.py to train 200 merges on a text file of your choice and print the ten longest tokens learned. Then tokenize a paragraph from a different domain (code, if you trained on prose) and compare tokens per word. Name the three most "wasteful" splits you see.