The next-token game

1.1Text becomes tokens

The game is played with words, but the machine does not see words. It sees tokens: chunks of text drawn from a fixed list. Common words are usually one token. Rarer words get split. Punctuation, spaces, and pieces of code are tokens too. The list is decided once, before any training, and it is called the vocabulary.

text      "The detective looked at the body"
            │
            ▼  chop into vocabulary pieces
tokens    ["The", " detective", " looked", " at", " the", " body"]
            │
            ▼  look each piece up in the vocabulary
ids       [ 791,  31305,  7111,  520,  279,  2547 ]

Two things to notice. First, the leading space is part of the token: " the" and "the" are different entries. Second, the model only ever sees the integers on the last line. "The" is not a word to the model; it is entry number 791 in a list.

How large is that list? For current frontier models, on the order of one to two hundred thousand entries. Llama 3 uses 128,256 public. Closed labs do not always publish theirs, but tokenizers are usually shipped with the API and can be inspected, so the sizes are known to be in the same range inferred. Call it V ≈ 128k and hold that number; it matters in a moment.

How the chopping is decided, why it produces odd splits, and what that does to model behavior is Chapter 6. For now: text in, list of integers out.

1.2One function

Strip away everything, and a language model is this:

Read the picture as a Python signature, because that is what it is:

def model(context: list[int], theta) -> list[float]:
    # returns V numbers, one per vocabulary entry, each ≥ 0, summing to 1
    ...

Three facts about this function carry the whole book.

It never answers. It always spreads. Ask it to continue "2 + 2 =" and it does not return " 4". It returns a distribution in which " 4" has, say, probability 0.97, " four" 0.02, and the other 128k tokens share the remaining 0.01. A good model is one whose spread is sharp where the text is predictable and honestly wide where it is not. "The killer was still in the …" should be wide. "2 + 2 =" should be sharp.

It is the same function at every step. To produce a paragraph, you call it, pick a token from the distribution, append that token to the context, and call it again. Nothing inside changes between calls. The parameters θ are fixed during generation. The only thing that changes is the context, which grows by one token each time.

Everything the model knows is in θ. The function has no database, no lookup, no memory across calls. Whatever it has learned about detectives, arithmetic, or Python is encoded in those billions of numbers and expressed only through the shape of the distribution it returns. Part 1 shows how numbers can encode that. Part 2 and 3 show how they are set.

1.3From scores to probabilities

The picture above cheats slightly. The inside of the model, once we open it, will not naturally produce numbers that are non-negative and sum to one. It will produce scores: one number per vocabulary entry, any real value, positive or negative, with no constraint at all. A score of 4.1 for " room", 2.3 for " house", −7.8 for " banana". Higher means "more likely", but they are not probabilities yet. These raw scores are called logits.

So the model needs a last step that converts V unconstrained scores into a valid distribution. The conversion has to do four things:

1. Make every value positive. Probabilities cannot be negative.
2. Make them sum to 1.
3. Keep the order. If " room" scored higher than " house", it must end up more probable.
4. Be smooth. Small changes to a score should make small changes to the probabilities, because in Chapter 5 we will nudge scores by tiny amounts and need the output to move gently in response.

The function that does all four is softmax. It is two operations: raise e to each score, then divide by the total.

import math
z = [2.0, 1.0, 0.0, -1.0]
ex = [math.exp(v) for v in z]
p  = [v / sum(ex) for v in ex]
print([round(v, 3) for v in p], round(sum(p), 6))

[0.644, 0.237, 0.087, 0.032] 1.0

Now play with it. The widget below is the table above with sliders.

Temperature

You saw a temperature slider. It does one thing: divides every score by a number T before the exponential. T = 1 changes nothing. T = 0.5 doubles every score, which doubles every gap, which makes the winner win harder. T = 2 halves the gaps and flattens the distribution toward uniform. As T → 0 the distribution collapses onto the single top score; as T → ∞ it becomes flat.

Temperature is not part of the model and not learned. It is a knob applied at generation time to decide how adventurous the chip-placing should be. We return to it in §1.5 and properly in Chapter 7.

1.4The bill: loss as surprise

Back to the game. You placed your chips, the true word was revealed, and now you are charged. The rule for the charge is the single most important formula in this book, and it is short:

loss = −ln p(target)

Take the probability the model assigned to the token that actually came next. Take its natural log. Flip the sign. That is the bill.

Why this shape? Work through what it does at the edges.

This quantity has a name: surprise. It measures how unexpected the outcome was given what you predicted. Low probability, high surprise. And the loss for a whole text is just the surprise summed over every token in it.

Three reasons it is exactly this formula and not some other decreasing function of p.

1. Products become sums. The probability the model assigns to a whole sentence is the product of the probabilities it gave to each token in turn: p(t₁) · p(t₂ | t₁) · p(t₃ | t₁, t₂) · …. Products of small numbers vanish into floating-point noise after a few dozen tokens. Logs fix it: the log of the product is the sum of the logs, so the loss of a sentence is the sum of per-token losses, a number computers can hold and compare. This is the third property from the Math box, doing real work.

2. Hedging is rewarded honestly. Suppose you genuinely believe the next word is room with 60% and house with 40%. With this scoring rule, the bet that minimises your expected bill is to put exactly 60 chips on room and 40 on house. Not 100 on room. Not 50/50. Your true belief. A scoring rule with that property is called a proper scoring rule, and −ln p is the simplest one. It means a model trained to minimise this loss is pushed toward reporting its actual uncertainty, not toward bravado. This is why a well-trained model's probabilities tend to be meaningful and not just rankings.

3. Zero is forbidden. The infinite charge for p = 0 looks harsh, but it is the point. It means the model can never afford to rule anything out completely. Every token keeps a sliver of probability, however small. That sliver is what lets the model recover when the text takes a turn it did not expect.

Perplexity: the effective number of choices

Take the mean loss per token over a text, then undo the log: perplexity = emean loss. If the mean loss is 2.3 nats, perplexity is 10. Read that as: "on average, the model was as uncertain as if it were choosing uniformly among 10 equally likely tokens." A perplexity of 1 is a model that is always certain and always right. A perplexity of 128k is a model that has learned nothing and spreads its chips evenly over the whole vocabulary. Frontier models on ordinary English land in the low single digits inferred: from published pretraining curves of open models, whose loss on held-out web text sits below 2 nats, and from the general shape of scaling results in Chapter 8.

Look at the last row of that widget. The model gave mat only 12% (it preferred rug, perhaps). That single token costs more than the other five combined. Loss is dominated by the tokens the model got most wrong, which is exactly where you would want a learner's attention to go. Chapter 5 shows that this is not a metaphor: the size of the loss on a token directly sets how hard the parameters get pushed because of it.

1.5Playing the game: generation

You have the function and the scoring rule. Now generate text. The loop is four lines long and it is the same loop for every language model ever deployed:

context = tokenize(prompt)
while not done:
    p = model(context, theta)     # V probabilities
    t = choose(p)                 # one token id
    context.append(t)

The only choice you have is inside choose. Two families:

• Greedy: always take the highest-probability token. Deterministic. Tends to produce flat, repetitive text, because the single most likely next token, chosen every time, drifts toward the blandest continuation.
• Sampling: draw a token at random, with the model's probabilities as the odds. A token with p = 0.3 is picked 30% of the time. Text varies from run to run. Temperature, from §1.3, is applied to the scores before sampling to make it tamer or wilder.

Play it. The widget holds a toy model with a nine-token vocabulary. It has a real limitation that you should notice: it decides its distribution from the last token only. A real model conditions on the entire context. That gap, from "last token" to "everything so far", is exactly what Chapter 3 fills with attention.

Try three things. Press Take the most likely eight times and watch it settle into a loop. Reset, and press Sample eight times: same model, different sentence. Then push the temperature to 3 and sample: the bars flatten and the sentences turn to nonsense, because the model's preferences are being ignored. Push it to 0.2 and sampling becomes greedy in disguise.

Nothing about the model changed in any of those runs. Only choose did. Keep that separation clean in your head: the model produces the distribution; the sampler picks from it. Many things people attribute to "the model" (creativity, repetitiveness, randomness) live in the sampler.

1.6Two loops, one function

There are two activities in the life of a model, and the same function sits in the middle of both.

Training is the game played for bills. Feed the model a slice of real text, ask it for the distribution at every position, bill it for every true next token, then adjust θ a little so the same bills would have been slightly lower. Repeat over trillions of tokens. The "adjust a little" step is Chapter 5; it is where calculus enters, and it is the only place anything is learned.

Inference is the game played for real. Feed the model a prompt, sample, append, repeat. θ is frozen. When you use a chat assistant, this is all that happens on the other end: the loop of §1.5, on a very large θ, very fast, for you and a few million other people at once. How that is made fast and cheap is Chapter 19.

A common confusion, cleared now: the model does not learn from your conversation. Nothing you type changes θ. It changes the context, and a different context gives a different distribution, which can look like learning. It is not. The red arrow is not there.

1.7The Lab and Beacon

Time to introduce the model this book follows. It does not exist. Call it Beacon, built by an organisation we will call the Lab. Beacon stands in for a current frontier model, and every part of the book asks what the Lab does to it at that stage: how the corpus is built, how the run is planned, how the assistant behavior is shaped, how it is served. Where the real labs have published, Beacon follows their published numbers, marked public. Where they have not, the book says so.

Here is what Chapter 1 already tells you about Beacon.

reply length          1,000 tokens
distributions         1,000  (one per generated token)
numbers per dist.   128,000
numbers computed    128,000,000   ≈ 1.3 × 10⁸
numbers kept              1,000   (the sampled token ids)

1.8Builder's bench: Dispatch says hello

The second thread of this book is a product you build on a frontier API, one layer per chapter. It is called Dispatch: an on-call assistant for the engineers who run Postbox, a fictional notification-delivery service. By Part 4 Dispatch will read runbooks, call tools, triage incidents, and be measured before every release. Today it says hello.

The first call is deliberately plain, so that you can see the game of this chapter underneath it. A system prompt sets the context. A user message extends it. The model plays the next-token game from there until it decides to stop.

# code/ch01/dispatch_hello.py
import anthropic

client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

SYSTEM = """You are Dispatch, the on-call assistant for Postbox, a notification
delivery service. Answer briefly and concretely. If you do not know
something about Postbox, say so."""

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    system=SYSTEM,
    messages=[{
        "role": "user",
        "content": "Pager went off: 'push-queue depth > 50k'. Where do I start?",
    }],
)
for block in response.content:
    if block.type == "text":
        print(block.text)
print("stop_reason:", response.stop_reason)
print("output tokens:", response.usage.output_tokens)

(example output — the exact text differs per run, because each token is sampled)
Start with three checks: (1) is the queue growing or draining — compare depth over the
last 15 minutes; (2) are consumers healthy — look at push-worker error rate and restarts;
(3) did volume spike upstream — check inbound event rate. I don't have Postbox's dashboards,
so tell me which of these you can see and I'll narrow it down.
stop_reason: end_turn
output tokens: 94

Map that back to the chapter. The system prompt and your message became roughly 80 tokens of context. The model then played 94 rounds of the game: 94 distributions over 128k tokens, 94 samples, 94 appends. On the 94th round it sampled a special end-of-turn token, and the loop stopped. stop_reason: end_turn is the sampler reporting that the model itself chose to end. The other common value, max_tokens, means you cut it off.

Every number in response.usage is a count of rounds of the game. Every price on every provider's pricing page is a price per round.

Seeing the distribution

Most frontier APIs return the sampled token, not the distribution it was drawn from. But you can see the distribution's shadow: ask for a single token, many times, and tally what comes back. If the model spreads its chips, the tally spreads.

# code/ch01/first_token_tally.py
import collections
import anthropic

client = anthropic.Anthropic()
prompt = "Complete this sentence with one word: The detective knew the killer was still in the"

tally = collections.Counter()
for _ in range(60):
    r = client.messages.create(
        model="claude-haiku-4-5",   # a model that still accepts a temperature parameter
        max_tokens=3,
        temperature=1.0,           # sample with the model's own odds
        messages=[{"role": "user", "content": prompt}],
    )
    first = r.content[0].text.strip().split()[0].strip(".,").lower()
    tally[first] += 1

for word, n in tally.most_common():
    print(f"{word:<10} {n:>3}  {'█' * n}")

(example output — a tally of 60 samples; yours will differ in the details)
room        31  ███████████████████████████████
house       14  ██████████████
building     8  ████████
area         4  ████
mansion      2  ██
vicinity     1  █

That histogram is the chip layout, observed from outside. Run it again at temperature=0.2 and the tally collapses to almost all room. Run it at temperature=1.5 if the model allows it and the tail grows.

Two notes on the frontier reality here. The newest models from some providers, including the current top models on the Claude API, no longer accept a temperature parameter at all public: the lab has fixed the sampling policy. The knob has moved from your side of the API to theirs. Second, if a provider does expose token probabilities directly (some do, under names like logprobs), you can skip the tally and read the distribution straight off. Either way the object underneath is the same: V numbers summing to one.