When the answer can be checked, the model can teach itself to think in tokens.
Chapter 14 taught the base model to behave like an assistant by imitating examples. Chapter 15 taught it to prefer what people prefer, using a learned reward model as the judge. This chapter removes the judge. For a large class of problems, mathematics, code, anything with a test, the answer can be checked by a program, and a checkable answer is a reward that cannot be argued with. Training on that reward, with the model writing its own long attempts, is how the reasoning models of 2024 and 2025 were made. It also reopens a limit from Chapter 4: a transformer's depth is fixed, so the only way it can think longer is to write more tokens. Chapter 17 asks what this does to safety; Chapter 18 how any of it is measured.
The question this chapter answers: how does a model learn to reason from nothing but a checker, and why does "thinking" look like writing?
A student preparing for a maths olympiad, alone, with a stack of past papers and the answer key. No teacher. No worked solutions. For each problem she writes an attempt, checks the final number against the key, and marks it right or wrong. Nothing else is graded: not the neatness, not the method, not whether the reasoning was the "intended" one.
What changes over months? Not her knowledge of arithmetic; she had that. Her habits change. She learns to write out intermediate steps because attempts with steps come out right more often than attempts that jump to an answer. She learns to try a second route when the first looks off, because attempts that checked themselves scored better. She learns to spend longer on hard problems and less on easy ones. None of this was in the key. It emerged because some habits produced ticks and others produced crosses, and she kept the habits that produced ticks.
Now give her a very fast clock and let her write eight attempts per problem instead of one, and compare them to each other rather than to any absolute standard. The attempts that beat their siblings get reinforced; the ones that lose get suppressed. That is reinforcement learning from verifiable rewards, and the eight-sibling comparison is GRPO.
| In the picture | In the machine | The word we will use |
|---|---|---|
| The answer key | A program that checks a final answer (equality, a unit test, a compiler) | verifier; the reward it gives is verifiable |
| Writing an attempt | Sampling a full generation from the current model at temperature > 0 | rollout, a sample from the policy |
| Eight attempts at one problem | G rollouts for the same prompt | a group |
| "Better than my other attempts" | Reward minus the group mean, divided by the group spread | group-relative advantage |
| Keeping habits that earned ticks | Raising the probability of every token in above-average attempts, lowering it in below-average ones | policy gradient; GRPO is the specific recipe |
| Not drifting into nonsense while chasing ticks | A penalty on how far the model has moved from where it started | KL penalty to the reference model |
| Writing steps because steps help | Long generations before the answer, learned because they raise reward | chain of thought, reasoning tokens |
| Spending longer on harder problems | Generating more tokens, or more attempts, at answer time | test-time compute |
Chapter 15's reward model was a neural network trained to guess what a human would prefer. It was useful and it was gameable: a policy optimised hard enough against it found responses the reward model liked and humans did not. Every guard in that chapter, the KL penalty, early stopping, fresh preference rounds, was there to slow the gaming down.
A verifiable reward has no such weakness in the reward itself. If the task is "compute 17 × 24", the reward is 1 if the final number is 408 and 0 otherwise. If the task is "write a function that passes these tests", the reward is the test runner's exit code. If the task is a formal proof, the reward is the proof checker. The reward can still be gamed at the edges (the model can learn to print an answer without a valid method, or exploit a weak test suite), but the core signal is exact, free, and unlimited in quantity. That is the difference between a few hundred thousand human comparisons and millions of machine-checked problems.
The catch is coverage. Most of what an assistant does has no checker: there is no unit test for a good explanation or a kind refusal. So the labs use verifiable rewards where they exist, mostly maths, code, and structured formats, and rely on Chapter 15's methods elsewhere. Published pipelines combine both in the same run public (DeepSeek-R1's final stage; Llama 3's mixed post-training data; Tulu 3's "RL with verifiable rewards" stage).
where the reward comes from
human preference (ch 15) reward model R(x, y) learned, gameable, covers everything
verifiable (this chapter) rule: check(y) ∈ {0, 1} exact, cheap, covers only checkable tasks
format rule: "answer inside \boxed{}" exact; teaches the shape of an answer
Recall PPO from Chapter 15. To decide whether a sampled response was good, it compared the response's reward against a value model, a second network of the same size as the policy, trained to predict the expected reward. That comparison, reward minus expectation, is the advantage: it says whether this sample did better than the policy usually does here. The value model was the most expensive and least stable part of the pipeline.
Group Relative Policy Optimisation drops it. For each prompt, sample G responses instead of one. Score them all. Use the group itself as the expectation: a sample's advantage is its reward minus the group's mean reward, divided by the group's standard deviation.
for one prompt q, sample G responses o₁ … o_G from the old policy π_old rewards r₁ … r_G from the verifier advantages Aᵢ = (rᵢ − mean(r)) / std(r) same Aᵢ for every token of oᵢ per token t of response i, the probability ratio ρᵢ,ₜ = π_θ(oᵢ,ₜ | q, oᵢ,<ₜ) / π_old(oᵢ,ₜ | q, oᵢ,<ₜ) objective (maximise) J(θ) = mean over i, t of min( ρᵢ,ₜ · Aᵢ , clip(ρᵢ,ₜ, 1−ε, 1+ε) · Aᵢ ) − β · KL(π_θ ‖ π_ref)
Read it in pieces. ρ is how much more (or less) likely the new policy makes a token than the policy that sampled it; at the first update step it is exactly 1. Multiplying by A means: if this response beat its siblings, reward the update for raising its tokens' probability; if it lost, reward lowering them. The clip and the min are PPO's brake: once a token's ratio has moved past 1 ± ε in the helpful direction, the objective stops rewarding further movement, so one batch cannot fling the policy. The KL term is the leash to the reference model from Chapter 15, and β sets its length. DeepSeek estimates the KL per token with an unbiased estimator rather than adding it to the reward public.
Chapter 5 tells you what the gradient of this is: for every token in every response, a push on the logits of size proportional to Aᵢ, delivered by backpropagation through the same graph as pretraining. GRPO is the pretraining loop with a different sign on each token.
One prompt, four samples, a rule-based verifier.
prompt "What is 17 × 24?" answer key: 408 o₁ "17×24 = 17×20 + 17×4 = 340 + 68 = 408" r₁ = 1 o₂ "17×24 = 17×25 − 17 = 425 − 17 = 418" r₂ = 0 (arithmetic slip) o₃ "24×17 = 24×10 + 24×7 = 240 + 158 = 398" r₃ = 0 (24×7 ≠ 158) o₄ "17×24: 17×24 = 408" r₄ = 1 (right, no method) mean = 0.5 std = 0.5 A = [ (1−.5)/.5, (0−.5)/.5, (0−.5)/.5, (1−.5)/.5 ] = [ +1, −1, −1, +1 ] what the update does every token of o₁ and o₄ probability pushed up (ratio may rise to at most 1+ε = 1.2 before the clip) every token of o₂ and o₃ probability pushed down (ratio may fall to at most 1−ε = 0.8)
$ python code/ch16/grpo_by_hand.pyrewards [1. 0. 0. 1.] mean, std 0.5 0.5 advantages [ 1. -1. -1. 1.] ratio 0.8: objective for A=+1 → +0.80 for A=−1 → -0.80 ratio 1.0: objective for A=+1 → +1.00 for A=−1 → -1.00 ratio 1.1: objective for A=+1 → +1.10 for A=−1 → -1.10 ratio 1.3: objective for A=+1 → +1.20 for A=−1 → -1.30
Two things to notice. First, o₄ got the same push as o₁ although it showed no work: the verifier only reads the final number. Whether "show your work" gets reinforced depends on whether showing it makes the final number right more often, which for a base model it does; the habit is learned instrumentally, exactly as the student's was. Second, the last line: with A = +1 the objective is capped at 1.20 once the ratio passes 1.2, so a token that has already been boosted stops earning; with A = −1 the objective keeps falling past 0.8 (the min picks the unclipped, worse value), so the brake is asymmetric by design: it stops you overshooting the good direction, never the bad one.
The last case in that caption is the practical heart of GRPO: a prompt the model always gets right teaches nothing, and so does a prompt it always gets wrong. Useful gradient comes from prompts of the right difficulty, where the group is mixed. Published recipes filter prompts by the base model's pass rate for exactly this reason public, and the curriculum shifts as the model improves.
Chapter 4 ended on a constraint that seemed like trivia at the time: a token passes through each block exactly once, there is no loop, and the depth L is fixed. So a forward pass does a fixed amount of computation no matter how hard the question. "What is 2 + 2?" and "prove this lemma" get the same 126 blocks. There is no way, inside one pass, to think longer.
There is a way outside it. Every generated token is another full pass, and every generated token goes into the context of the next one. If the model writes "17 × 20 = 340" before writing the answer, then when it comes to write the answer, the value 340 is in its context, available to attention, and does not need to be recomputed from scratch. The written intermediate step is external memory and extra depth at once. A chain of a thousand reasoning tokens is a thousand extra forward passes, each able to read the results of all the previous ones.
This was known before reinforcement learning made it a training target. Chain-of-thought prompting, asking the model to "think step by step" or showing it worked examples, raised accuracy on arithmetic and logic problems sharply in models that had never been trained to do it public (Wei et al., 2022). The steps were already latent in the pretraining data, which contains a great deal of worked-out reasoning; prompting elicited the habit. What reinforcement learning adds is selection pressure: reasoning traces that lead to verified answers are reinforced, so the habit becomes the default, gets longer, and acquires behaviours that no prompt asked for.
The clearest public account of what emerges is the DeepSeek-R1 report public. Starting from a pretrained base model with no supervised examples of reasoning, only GRPO on maths and code with a rule-based reward for correctness plus a format reward for putting the thinking between tags, the model's average response length grew steadily over thousands of steps, from a few hundred tokens to many thousands, and its accuracy on a hard maths benchmark rose from about 16% to over 70% as it did. Along the way the traces developed re-checking ("wait, let me verify"), backtracking, and trying alternative approaches. The report calls one such moment, where the model interrupts itself to reconsider, an "aha moment"; nobody wrote it into the training data. The same report notes the cost of pure RL: the traces were hard to read and mixed languages, which is why the final recipe adds a small supervised stage first (§16.5).
Once thinking is writing, there are two dials at answer time that did not exist before, both of which trade compute for accuracy after training is over.
Longer thinking. Let the model generate more reasoning tokens before it commits. Published curves for reasoning models show accuracy on hard benchmarks rising roughly linearly with the logarithm of tokens spent thinking, over several orders of magnitude public (the OpenAI o1 announcement showed this shape; internals of that model are unknown). The shape matters: doubling the thinking budget buys a fixed increment, so the last doubling costs as much as all the previous ones and earns the same as any of them.
More attempts. Sample the whole problem n times and combine. Two combining rules, with very different needs:
1 − (1−p)ⁿ.$ python code/ch16/majority_vote.pyper-sample accuracy p = 0.3, wrong answers spread over m = 1 values n= 1 majority 0.292 best-of-n (perfect verifier) 0.300 n= 5 majority 0.162 best-of-n (perfect verifier) 0.832 n= 17 majority 0.039 best-of-n (perfect verifier) 0.998 n= 65 majority 0.000 best-of-n (perfect verifier) 1.000 per-sample accuracy p = 0.3, wrong answers spread over m = 8 values n= 1 majority 0.299 best-of-n (perfect verifier) 0.300 n= 5 majority 0.447 best-of-n (perfect verifier) 0.832 n= 17 majority 0.745 best-of-n (perfect verifier) 0.998 n= 65 majority 0.990 best-of-n (perfect verifier) 1.000 per-sample accuracy p = 0.6, wrong answers spread over m = 8 values n= 1 majority 0.601 best-of-n (perfect verifier) 0.600 n= 5 majority 0.878 best-of-n (perfect verifier) 0.990 n= 17 majority 0.999 best-of-n (perfect verifier) 1.000 n= 65 majority 1.000 best-of-n (perfect verifier) 1.000
Read the first block against the second. Same per-attempt accuracy, 30%. When every wrong attempt agrees on the same wrong answer, voting makes things worse: the wrong answer wins the vote. When wrong attempts are scattered over eight different wrong answers, the lone correct answer accumulates votes and 17 attempts reach 75%. Majority vote is a bet that errors are diverse and truth is unique; that bet pays on arithmetic and fails on a systematic misconception.
These dials feed straight back into training. Best-of-n with a verifier is not only an inference trick; it is how rejection-sampling data is made (sample many, keep the verified ones, fine-tune on them), which was the STaR recipe in 2022 public and is a stage of every published reasoning pipeline since. And a reasoning model that has been trained to think longer on hard problems is, in effect, choosing its own point on the test-time-compute curve per question.
The Lab does not publish its reasoning pipeline in full; no closed lab does unknown. But one frontier-class reasoning model was released with its recipe, and it is the reference for everything in this chapter public. Four stages, starting from the DeepSeek-V3 base model (671B total parameters, a mixture of experts; Chapter 12):
stage 0 R1-Zero (an experiment, released separately)
base model → GRPO with rule rewards (correct answer + thinking-tag format) → strong but unreadable
stage 1 cold start
a few thousand long, clean reasoning examples (curated, partly from R1-Zero outputs) → brief SFT
purpose: readable format and language consistency before RL, so RL starts from good habits
stage 2 reasoning RL
GRPO on maths, code, science, logic with verifiable rewards; plus a language-consistency reward
response length and accuracy climb together over thousands of steps
stage 3 rejection sampling + SFT
sample many responses from the stage-2 model; keep verified-correct, readable ones (≈600k reasoning)
add ≈200k non-reasoning examples (writing, QA, chat) from the V3 pipeline; SFT the base model on all of it
stage 4 RL for all scenarios
verifiable rewards for reasoning prompts + reward models (ch 15) for helpfulness and harmlessness
→ the released R1
distil the stage-3 data alone, used to SFT small open models (1.5B to 70B) → most of R1's reasoning at a fraction of the size
Three lessons the report states explicitly and that generalise public. First, pure RL from a base model works, which settles that reasoning need not be taught by example. Second, a small supervised cold start makes the result readable and the RL faster; the two methods are complements, not rivals. Third, for small models, distilling the traces of a large reasoner beat running RL on the small model directly: the search that discovers good reasoning is expensive, and its results transfer as data. Chapter 27 takes up that last point.
The report also records what did not work for them, which is rarer and more useful: a process reward model (scoring each step rather than the final answer) was hard to train, easy to hack, and did not help at scale; a tree search over reasoning steps, in the style of game-playing systems, did not scale because the "moves" in language are not a small, enumerable set public. Neither result rules those ideas out; both say that outcome rewards and plain sampling were enough to get here.
An outcome reward scores the final answer. A process reward scores each step. The case for process rewards is credit assignment: with an outcome reward, a twenty-step derivation that is right by luck gets every step reinforced, and a derivation that is perfect until a slip in step nineteen gets every step suppressed. Step-level scores would fix that, and a human-labelled dataset of step judgements exists and improved verification in one well-known study public (Lightman et al., 2023). The case against is cost and gaming: step labels are expensive, a learned step-scorer is a reward model with all of Chapter 15's weaknesses, and a policy will learn to write steps the scorer likes rather than steps that lead anywhere. As of the latest published work, outcome rewards with group-relative advantages carry most of the load, and process supervision remains an active question inferred from the R1 report and the subsequent open literature.
Verifiable rewards are exact, but the policy still optimises the letter of the check, not the spirit:
Frontier APIs expose reasoning as a distinct kind of output. On the Claude API the model thinks by default and the request can ask for a readable summary of that thinking to be returned alongside the answer public; the raw trace is not returned. The script asks a small time-arithmetic question and prints the summary block, the answer block, and the usage, which counts thinking tokens as output tokens because, as §16.3 explained, that is exactly what they are.
# code/ch16/thinking_observe.py
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-opus-5",
max_tokens=16000,
thinking={"type": "adaptive", "display": "summarized"}, # thinking is on by default; ask to see a summary
messages=[{"role": "user", "content": "A train leaves at 9:40 and arrives at 13:05 the same day. "
"How long is the journey, in minutes? Show only the number."}],
)
for block in response.content:
if block.type == "thinking":
print("[thinking summary]", block.thinking[:400], "…")
elif block.type == "text":
print("[answer]", block.text)
print("usage:", response.usage.input_tokens, "in,", response.usage.output_tokens, "out (thinking tokens are billed as output)")(example output — the summary text and token counts vary per run) [thinking summary] From 9:40 to 13:05: 9:40 to 13:00 is 3 hours 20 minutes, plus 5 minutes is 3 h 25 min. Convert: 3 × 60 + 25 = 205. Checking the other way, 13:05 − 9:40 = 3:25 → 205 minutes. … [answer] 205 usage: 47 in, 112 out (thinking tokens are billed as output)
Read the usage line against the answer. The answer is one token, 205. The output count is over a hundred: the rest was thinking, and it was billed, because it was generated. What Dispatch will pay for when it reasons about an incident (Chapter 22) is this ratio, and the effort setting that the current API exposes is the knob on how far along §16.4's log curve to go public.
| Quantity | DeepSeek-R1 (public reference) | Evidence |
|---|---|---|
| Base model | DeepSeek-V3-Base, 671B total / 37B active (MoE) | public |
| RL algorithm | GRPO, rule-based rewards (accuracy + format), KL to reference | public |
| R1-Zero response length | grew from hundreds to ≈10k tokens over ≈8k RL steps | public (figure in the report; exact values read from the plot are inferred) |
| R1-Zero AIME 2024 pass@1 | 15.6% → 71.0% (86.7% with majority vote over 64 samples) | public |
| Rejection-sampling SFT set | ≈600k reasoning + ≈200k non-reasoning samples | public |
| Distilled models | Qwen and Llama bases, 1.5B to 70B, SFT only on the 800k samples | public |
| Closed frontier reasoning models | RL on verifiable rewards is the consensus assumption; group sizes, prompt counts, reward details, and thinking budgets are not disclosed | inferred / unknown |
What does one GRPO step cost, compared with one pretraining step? A pretraining step (Chapter 7) is one forward-backward over a batch of fixed text: about 6N per token. An RL step must first generate its batch, one token at a time, with the KV cache, which is Chapter 7's decode loop and is memory-bound; then verify; then do a forward-backward on what it generated; and a forward pass on the reference model for the KL term.
one RL step, illustrative Beacon-class numbers (37B active parameters as in the reference)
prompts per step 512
group size G 16 → 8,192 rollouts
tokens per rollout 4,000 (reasoning) → 33M generated tokens
generation 33M decode steps, batched; at ≈ 2N ops each ≈ 2.4 × 10¹⁸ ops, but memory-bound: minutes, not seconds
verification run 8,192 checkers (cheap for maths; a sandbox per rollout for code)
policy forward + backward 6N × 33M ≈ 7 × 10¹⁸ ops
reference forward (KL) 2N × 33M ≈ 2.4 × 10¹⁸ ops
⇒ per token of training signal, RL costs ≈ 2× pretraining in arithmetic and far more in wall-clock,
because a third of the work is sequential decoding. Serving infrastructure (ch 19) becomes training infrastructure.
This is why reasoning RL runs are measured in thousands of steps rather than a million, why they are done on top of a fully pretrained model rather than from scratch, and why the labs that are best at serving models have an advantage at training reasoners: the bottleneck is the same decode loop.
Use G = 1. No group to compare against; the advantage is undefined (standard deviation of one number). You would need a value model to supply the expectation, which is PPO, with its second full-size network and its instability. The group is what makes the critic unnecessary.
Drop the clip. A single batch where one rare token got a large advantage can multiply that token's probability many-fold in one step; the next batch is sampled from the wrecked policy and cannot recover. The clip bounds how far any token can move per step to 1 ± ε, which is a per-token learning-rate cap; PPO introduced it for exactly this reason and every descendant keeps it.
Drop the KL leash (β = 0) and train long enough. The policy sharpens onto whatever the verifier likes, loses the diversity that made group comparison informative, and drifts on everything the verifier does not measure: readability, language, refusals, everything Chapter 14 and 15 built. R1-Zero's mixed-language traces are the mild version. The widget above shows the severe version in six answers.
Train only on prompts the model already solves. Every group is all-correct, every advantage is zero, every gradient is zero. Compute is spent, nothing changes. Difficulty filtering is not an optimisation; without it the method does nothing.
Reward the reasoning text instead of the final answer, with a learned scorer. You have rebuilt Chapter 15's reward model, and the policy will find reasoning that the scorer likes. Verifiable rewards are valuable precisely because they only look at what can be checked; extend them to what cannot, and you inherit the problems of learned judges.
Cap thinking at 200 tokens. The model cannot reach the answers that needed long chains, so long chains are never verified, so they are never reinforced. It learns to be a fast guesser. The log-linear curve of §16.4 is only available if the budget during training allowed the model to climb it.
Say it back. Where an answer can be checked by a program, the check is a reward that is exact, free, and unlimited, and Chapter 15's learned judge is not needed. The model is trained by sampling a group of full attempts at each prompt, scoring them with the verifier, and computing each attempt's advantage as its reward relative to its own group, so no value model is needed either. Every token of an above-average attempt is pushed up and every token of a below-average one pushed down, through the same backpropagation as pretraining, with a clip on how far any token may move per step and a KL leash to the reference model. Prompts the model always solves or always fails teach nothing, so difficulty is curated. Because a transformer's depth is fixed, the only way to compute more on a hard problem is to write more tokens, and reinforcement learning selects for exactly that: reasoning traces grow longer, acquire self-checks and backtracking nobody wrote, and accuracy climbs roughly with the logarithm of tokens spent. At answer time the same lever is available as test-time compute: think longer, or sample many attempts and vote or verify. The public recipe is a short supervised cold start, reasoning RL, rejection-sampled SFT on verified traces, then RL for everything, with the traces distilled into small models as data. The failure modes are the letter-versus-spirit ones: length inflation, weak checkers, format without substance, drift away from readability, and they are held in check by the leash, hidden tests, and length-aware rewards. Beacon's reasoning is billed as output tokens because that is what it is.
[1, 1, 0, 0, 0]. Compute the mean, the standard deviation, and the five advantages. Now a sixth attempt is added with reward 1; recompute. Explain in one sentence why the two correct attempts' advantages fell when a third correct one arrived, and what that implies about which prompts are worth sampling.G = 8, and 6,000 tokens per rollout. Compute the generated tokens per step, the arithmetic for policy forward-backward and reference forward using the 6N and 2N rules, and the decode time if generation runs at 2,000 tokens per second per replica across 64 replicas. Then compare the step's total token count with one 16M-token pretraining step and say which is the bottleneck and why.majority_vote.py: add a third combining rule, "weighted vote", where each attempt's vote counts 1 + 0.5·(its length in steps), with lengths drawn randomly, and correct attempts are on average one step longer. Plot accuracy versus n for all three rules at p = 0.3, m = 8. Then break the assumption (make wrong attempts longer) and describe what happened as a one-line instance of reward hacking.