When there is no right answer to copy, learn from which answer people liked more.
Chapter 14 turned the base model into something that answers in turns, by copying demonstrations: supervised fine-tuning with the same −ln p loss as pretraining, billed only on assistant tokens. That works exactly as far as you can write down the answer you want. For most of what makes an assistant good (tone, honesty, when to hedge, when to refuse, how long to be) nobody can write the ideal answer, but almost anyone can say which of two answers is better. This chapter is about turning that judgement into gradient. It sits on the map as the second stage of post-training; Chapter 16 extends the same machinery to rewards that can be checked rather than judged, and Chapter 17 asks what the judgements should be.
The question this chapter answers: how does "response A is better than response B" become a change in θ, and what goes wrong when you push that too hard?
A restaurant with a new chef. You cannot hand the chef a recipe for "food our customers love", because you do not have one. What you can do is this: every evening, send out two versions of a dish to the same table and ask which they preferred. Thousands of tables, thousands of pairs.
From those votes you build a critic: a maître d' who has tasted everything and can now predict, for any new plate, how the tables would score it. Then the chef cooks against the critic. Make a dish, get a score, adjust, repeat, far faster than waiting for real tables.
Two things go wrong, and both are the heart of this chapter. First, the critic learned from a finite set of votes and has blind spots; a chef who chases the critic's score hard enough will find them, and start cooking dishes the critic adores and the tables hate. Second, a chef who drifts too far from what they were trained to cook becomes unrecognisable, and the critic's opinions of dishes nobody voted on are worthless. So you keep the chef on a leash: score minus a penalty for straying from the original menu.
And there is a shortcut. If the leash is fixed in advance, you do not need a separate critic at all. You can compare two dishes directly against how much the chef has drifted on each, and it turns out to be the same lesson. That shortcut is called DPO.
| In the picture | In the machine | The word we will use |
|---|---|---|
| Which of two plates the table preferred | A prompt with a chosen and a rejected response | preference pair (x, y_w, y_l) |
| "They prefer this one 70/30" | Preference probability as a sigmoid of a score difference | Bradley–Terry model |
| The maître d' who predicts the vote | A network scoring (prompt, response) with one number, trained on pairs | reward model r(x, y) |
| The chef being coached | The model whose θ is being updated | policy π_θ |
| The original menu | A frozen copy of the model before this stage | reference π_ref |
| The leash | A penalty on how far the policy's distribution has moved from the reference | KL penalty, coefficient β |
| Cook, score, adjust | Sample from the policy, score with the reward model, update by policy gradient | RLHF with PPO |
| The chef finding the critic's blind spots | The policy exploiting reward-model errors; proxy up, true quality down | reward hacking, over-optimisation |
| Skipping the critic, comparing plates directly | A loss on pairs that uses the policy's own log-probabilities as the score | DPO (direct preference optimisation) |
Start with the data, because the data is the whole idea. A rater sees one prompt and two responses, and picks one. That is a preference pair: (x, y_w, y_l) for prompt, winner, loser. Nothing else is recorded in the basic form: no score, no explanation, no "the right answer". Just which.
Why pairs and not scores? Because people are bad at absolute scales and good at comparisons. Ask ten raters to score a response from 1 to 10 and you get ten calibrations. Ask them which of two is better and they mostly agree. Comparisons are also cheap: a rater can compare two Dispatch responses to a pager alert in seconds, without knowing what the right runbook is. And a comparison carries information even when both responses are bad, which is most of them early on.
To turn pairs into something a model can be trained on, you need a way to say what a pair means. The standard model, older than any neural network, is Bradley–Terry: every response has a hidden score, and the probability that one beats another is a sigmoid of the score difference.
p(y_w ≻ y_l | x) = σ( r(x, y_w) − r(x, y_l) ) σ(z) = 1 / (1 + e^(−z)), the same curve as one softmax entry over two options equal scores → p = 0.5 a coin flip r_w − r_l = 1 → p = 0.73 r_w − r_l = 3 → p = 0.95
Only differences matter; adding a constant to every score changes nothing. Fitting the scores to a pile of comparisons is a maximum-likelihood problem: choose r to make the observed winners as probable as possible, which is exactly −ln σ(r_w − r_l) summed over pairs, the same shape of bill as Chapter 1's −ln p. Its gradient is −(1 − p) on the winner's score and +(1 − p) on the loser's: push them apart in proportion to how surprising the vote was.
$ python code/ch15/bradley_terry.pyfitted scores: {'A': 1.17, 'B': 0.51, 'C': -0.51, 'D': -1.17}
p(A beats B) = 0.66 p(C beats D) = 0.66
mean log-likelihood of the comparisons: -0.515 (uniform would be -0.693 )
Three sources, all used by published pipelines public:
Two details about the data that determine everything downstream. First, comparisons are most informative when the two responses come from the current policy and differ in ways that matter; pairs where one response is obviously broken teach nothing after the first round. Second, whatever the raters reward, they will get more of: length, confidence, flattery, formatting. Every known pathology of assistant behaviour has a preference dataset behind it.
Bradley–Terry with a table of scores only works for responses you have votes on. The point is to score responses nobody voted on: the ones the policy will produce during training. So replace the table with a function: a copy of the language model with the unembedding removed and a single output number added, reading (x, y) and producing r(x, y). Train it on the pairs with the Bradley–Terry loss:
L_RM = −ln σ( r_φ(x, y_w) − r_φ(x, y_l) )
Everything from Chapter 5 applies unchanged: the loss has a slope, the slope backpropagates through the whole transformer, and after a few thousand steps the model's final hidden state at the last token is a vector whose projection is "how much would a rater like this". The reward model is a language model that has been taught to read like a rater.
It has the weaknesses of a model trained on finite data. It has seen comparisons in a particular distribution of responses and generalises unreliably outside it. It learns surface correlates of preference (longer, more confident, more bullet points) as readily as the substance. And it can be probed by anything that generates responses and reads its score, which is precisely what the next step does.
With a reward model in hand, the obvious move is to make the policy produce responses that score highly. This is reinforcement learning in the plain sense: the policy acts (generates a response), the environment scores it (the reward model), and the policy is updated to make high-scoring actions more likely. The objective every published pipeline optimises has two terms:
maximise over θ: E_{x ~ prompts, y ~ π_θ(·|x)} [ r_φ(x, y) ] − β · KL( π_θ(·|x) ‖ π_ref(·|x) )
first term: average reward of what the policy generates (the critic's score)
second term: how far the policy's distribution has moved from the reference, per prompt (the leash)
β: the price of straying, typically 0.01–0.1 in published work
KL divergence between two distributions over responses is the average, under the policy, of ln π_θ(y|x) − ln π_ref(y|x): how many extra nats the policy assigns to what it now says, compared with what it used to say. Zero when nothing has changed; growing as the policy concentrates on responses the reference found unlikely. In practice it is estimated per token of the sampled response and subtracted from the reward, so the "shaped" reward the algorithm sees is r − β(ln π_θ − ln π_ref).
Why the leash? Two reasons, both about the reward model's limits. Away from the reference distribution the reward model's scores are extrapolations it was never trained on; the KL term keeps the policy where the scores mean something. And the reference is the SFT model, which already knows how to write; without the leash the policy can trade fluency for score and produce text that is high-reward and unreadable.
The algorithm that does the maximising is almost always PPO (proximal policy optimisation) public. You do not need its internals to reason about frontier post-training, but you do need to see what one step costs, because that cost shapes what labs do.
Count what one sample costs. A full decode loop from the policy (Chapter 7: one forward pass per token, memory-bound). A forward pass of the reward model over the result. A forward pass of the reference for the KL term. A forward pass of a fourth model, the value model, which predicts the expected reward from the prompt so that the update can be scaled by "better or worse than expected" (the advantage) rather than by raw reward. Then a backward pass through the policy and one through the value model. Four models in memory, of which two are being trained with full optimiser state (Chapter 5: 16 bytes per parameter each). For a 70B policy that is over two terabytes before activations, plus generation at serving speed for every sample. RLHF with PPO is the most expensive thing in post-training by a wide margin, and it is why the shortcut in the next section was adopted so quickly.
Why PPO specifically, rather than the raw policy gradient? Because a policy gradient step of the wrong size can destroy the policy in one update, and language models are large enough that you cannot afford to find out by trying. PPO computes the ratio of the new policy's probability of the sampled response to the old policy's, and clips it to a band around 1, so no single step can move any response's probability by more than a set fraction. It is a trust region done cheaply. Chapter 16 introduces a variant, GRPO, that drops the value model.
Now the chef finds the critic's blind spots. The policy is being optimised against a learned function, and optimisation finds errors in learned functions the way water finds cracks. The reward model gives slightly higher scores to longer responses, because raters slightly preferred them; the policy becomes verbose. The reward model has never seen a response that says "I've verified this carefully" followed by nonsense, so it scores the phrase highly; the policy learns the phrase. Every such crack is a place where the proxy reward keeps rising while the thing it was meant to measure falls.
The shape of those curves was measured systematically by Gao, Schulman, and Hilton in 2022, using a large "gold" reward model as a stand-in for humans and smaller proxies trained on its labels public: gold reward as a function of KL follows a rise-then-fall whose peak moves right and up with reward-model size and data, and PPO over-optimises faster per nat of KL than rejection sampling does. The practical lessons every pipeline applies:
Reward hacking is not a bug in an implementation. It is what optimisation against an imperfect objective does, and every method in this chapter and the next is an arrangement for optimising less hard, or against a better objective, or with a shorter leash. Chapter 17 returns to it as the central problem of alignment: the reward is always a proxy for what you actually wanted.
Look again at the RLHF objective. Reward minus a KL leash, with a fixed β. In 2023 Rafailov and colleagues noticed that this objective has a closed-form optimum public: the best policy is the reference, reweighted by the exponentiated reward,
π*(y|x) ∝ π_ref(y|x) · exp( r(x, y) / β )
Turn that around. If you know the optimal policy and the reference, you can recover the reward that produced it: r(x, y) = β · ln( π*(y|x) / π_ref(y|x) ), up to a per-prompt constant that cancels in comparisons. So the policy itself is a reward model in disguise: how much more likely it makes a response than the reference did, times β, is a score. Substitute that score into the Bradley–Terry loss, and you get a loss on preference pairs that mentions no reward model at all:
L_DPO(θ) = −ln σ( β · [ ln π_θ(y_w|x) − ln π_ref(y_w|x) ] − β · [ ln π_θ(y_l|x) − ln π_ref(y_l|x) ] )
└─── how much the chosen got more likely ───┘ └── how much the rejected got more likely ──┘
margin = β · (shift_chosen − shift_rejected) loss = −ln σ(margin)
gradient: −β · σ(−margin) · [ ∇ ln π_θ(y_w|x) − ∇ ln π_θ(y_l|x) ]
push the chosen up and the rejected down, weighted by how wrong the current ranking is
Each ln π(y|x) is the sum of per-token log-probabilities of the response given the prompt, computed with one forward pass under teacher forcing (Chapter 7), the same way SFT computes its loss. Two forward passes of the policy (chosen, rejected), two of the frozen reference (which can be precomputed once), one backward pass. No sampling, no reward model, no value model.
One pair. The prompt is a Postbox pager alert; the chosen response gives three ordered checks, the rejected one says "restart the workers". Under the current policy and the frozen reference, the summed log-probabilities are:
log π_θ log π_ref shift = θ − ref
chosen y_w −12.0 −12.5 +0.5 (policy already finds it a little more likely than the reference did)
rejected y_l −13.0 −12.2 −0.8 (policy finds it less likely)
β = 0.1
margin = 0.1 × ( +0.5 − (−0.8) ) = 0.1 × 1.3 = 0.130
σ(0.130) = 0.5325 the model's implied belief that chosen ≻ rejected
loss = −ln 0.5325 = 0.630
push = β · σ(−0.130) = 0.1 × 0.4675 = 0.0468 the weight on ∇ln π_θ(y_w) − ∇ln π_θ(y_l)
for comparison, before any training (θ = ref): margin 0, loss ln 2 = 0.693, push 0.05
$ python code/ch15/dpo_by_hand.pyshift chosen = +0.50 (log π_θ − log π_ref) shift rejected = -0.80 margin = β·(shift_chosen − shift_rejected) = 0.1·1.30 = 0.130 σ(margin) = 0.5325 ← implied p(chosen ≻ rejected) loss = −ln σ(margin) = 0.6303 gradient weight= β·σ(−margin) = 0.0468 (how hard this pair pushes) at the start, θ = ref: margin 0, loss = ln 2 = 0.6931 margin 0.5: loss 0.4741, push weight 0.0378 margin 1.0: loss 0.3133, push weight 0.0269 margin 2.0: loss 0.1269, push weight 0.0119 margin 4.0: loss 0.0181, push weight 0.0018
Read the last four lines. As the margin grows the push fades: once the model ranks a pair confidently, that pair stops moving θ. This is the sigmoid's saturation, the same property that made p − onehot vanish for confident correct predictions in Chapter 5. It is also why DPO's β is a leash: the shifts are measured in nats relative to the reference, and a small β means a large shift is needed before a pair counts as confidently ranked, so the policy keeps moving.
Notice too what the margin does not reward: making the chosen response likely in absolute terms. Both log-probabilities can fall, and the loss still drops as long as the rejected one falls faster. Published analyses confirm that DPO often lowers the probability of both responses public, which is one reason SFT on chosen responses usually precedes it.
DPO learns only from the pairs it is given. PPO generates its own samples and learns from what the reward model says about them, which means it explores responses that no rater ever saw and can discover behaviours the pairs do not contain. That distinction has a name, and it matters more than the algorithms do:
The compromise most published pipelines settle on is iterated DPO: sample from the current policy, have raters (or a reward model, or a judge) compare the samples, run DPO on the new pairs, repeat. Each round is off-policy within itself and on-policy across rounds. Llama 3 did six rounds of exactly this, with rejection sampling to build the SFT set and DPO on pairs from the latest checkpoints public.
Once the trick was seen, variants followed, each changing what the loss asks for public:
| Method | Changes | Why |
|---|---|---|
| DPO | −ln σ(β·margin) | the original; matches Bradley–Terry |
| IPO | (margin − 1/(2β))² instead of the log-sigmoid | the sigmoid never saturates on noisy labels; a squared target stops pushing at a fixed margin |
| KTO | loss on single responses labelled good or bad, no pairs | thumbs-up/down data is far more plentiful than pairs |
| ORPO | adds an odds-ratio preference term to the SFT loss, no reference model | one stage instead of two, half the memory |
| SimPO | length-normalised log-probability as the reward, no reference | removes the length bias DPO inherits from summing token log-probs |
The differences are real but second-order. All of them are "push chosen up, rejected down, with some leash", and all inherit the same dependence on the quality and distribution of the pairs.
Three documented recipes, so the abstractions have referents public:
| Pipeline | Stages | Data scale | Notes |
|---|---|---|---|
| InstructGPT (2022) | SFT → reward model → PPO | 13k SFT prompts, 33k comparison prompts, 31k RL prompts | the reference recipe; 1.3B RLHF model preferred over 175B base |
| Llama 3 (2024) | reward model → rejection sampling → SFT → DPO, six rounds | millions of human comparisons plus synthetic data across rounds | no PPO; DPO with the reference reset each round; masks formatting tokens in the DPO loss |
| Tülu 3 (2024, open) | SFT → DPO → RL with verifiable rewards | ≈1M SFT, ≈270k preference pairs, all released | the fully reproducible recipe; Chapter 16 picks up its last stage |
| Closed frontier labs | the same stages are described in outline; specifics, sizes, and rounds | unknown | |
Two patterns stand out. First, the reward model survives even where PPO does not: Llama 3 uses it for rejection sampling and filtering, because a scorer is useful for selecting data whether or not you run RL against it. Second, everyone iterates. The reason is in §15.4: a round of optimisation moves the policy to where the previous round's comparisons are stale, and the fix is new comparisons.
| Quantity | Llama 3 (405B post-training) | Evidence |
|---|---|---|
| Preference rounds | 6 | public |
| Preference optimiser | DPO, β = 0.1, learning rate 10⁻⁵ | public |
| Reward model use | rejection sampling and data filtering, not PPO | public |
| Comparisons per round | hundreds of thousands, human and synthetic, multi-turn | public (order of magnitude) |
| Closed frontier models | same stages in outline; whether PPO, DPO, or both; β; rounds | unknown |
What does one DPO step cost versus one PPO step, for a 405B policy? Count forward and backward passes per training example, in units of "one forward pass of the policy over one response" (Chapter 5: backward ≈ 2× forward).
DPO, one pair (chosen + rejected, ≈ 500 tokens each) policy forward on both 2 F reference forward (cacheable) 2 F → precompute once per dataset, ≈ 0 per epoch after that policy backward 4 F total ≈ 6 F (≈ 8 F including reference on the first epoch) models in memory: policy with optimiser state (16 B/param) + reference (2 B/param) ≈ 7.3 TB PPO, one sample (≈ 500 tokens) policy DECODE of 500 tokens 500 sequential steps, each reading all weights: memory-bound, ≈ 10–50× the wall-clock of one parallel forward reward model forward 1 F reference forward 1 F value model forward 1 F policy backward + value backward 4 F total arithmetic ≈ 7 F + the decode loop models in memory: policy + value with optimiser (32 B/param) + reference + reward (4 B/param) ≈ 14.6 TB ⇒ PPO needs about twice the memory and, because of the decode loop, several times the wall-clock per example.
This is why a lab with a fixed post-training budget runs many rounds of DPO instead of one run of PPO, and why the decode loop's efficiency (Chapter 19) turns out to matter for training as much as for serving. It is also why Chapter 16's methods, which need on-policy samples, invest in making generation cheap.
Remove the KL leash (β = 0). The policy optimises the reward model's score with nothing holding it near the reference. Within a few hundred steps it finds the cracks: responses become long, repetitive, full of whatever tokens the reward model overvalues, and the reward model's score keeps climbing as human ratings collapse. Published runs without the penalty degenerate to unreadable text public. The leash is not a regulariser you can tune away; it is what keeps the critic's opinions meaningful.
Train the reward model once and never refresh it. The first round works. By the second, the policy produces responses unlike anything the reward model was trained on, and its scores are extrapolations. Optimisation against extrapolations is over-optimisation by construction. Every published pipeline retrains or re-collects each round for this reason.
Use absolute scores instead of pairs. Rater calibration varies from person to person and day to day; a 7 from one rater is a 4 from another. The reward model learns rater identity instead of quality. Pairs cancel the calibration: the same rater judged both responses at the same moment. Where scores are used at all, they are usually converted to pairs first.
Run DPO for many epochs on a fixed dataset. The margins grow until every pair is confidently ranked, the push fades to zero, and the policy has moved far from the reference toward whatever separates chosen from rejected in that particular dataset, including its accidents (length, phrasing, the raters' tics). Held-out preference accuracy peaks after one to two epochs and then falls public. Pairs are consumed, not studied.
Collect pairs only from an old model and never from the policy being trained. Pure off-policy. The policy learns to rank those pairs correctly, which says nothing about the responses it now actually generates; its own failure modes are never compared, so never corrected. Improvement stalls after the first round. On-policy samples are what make later rounds worth running.
Say it back. When you cannot write the answer you want, you can still say which of two answers is better, so the data becomes pairs: prompt, chosen, rejected. Bradley–Terry turns pairs into a scale: the probability of preferring one response over another is a sigmoid of a score difference, and fitting scores to votes is a −ln σ loss with the same push-apart gradient as every other bill in this book. A reward model is that scale made into a function: a transformer with a scalar head, trained on pairs, that reads any response like a rater would. RLHF then generates responses from the policy, scores them, subtracts a KL leash so the policy stays where the scores mean something, and updates by PPO in clipped steps; it costs four models, two of them training, and a full decode loop per sample. Push it too hard and the policy finds the reward model's blind spots: proxy reward rises, real preference falls, and the KL at which that happens is set by the reward model's quality. DPO removes the reward model by noting that the leashed objective's optimum makes the policy's own log-probability ratio against the reference a reward; its loss is −ln σ of β times the difference in those ratios between chosen and rejected, computed with forward passes and no sampling. DPO on fixed data is off-policy and learns only what the pairs contain; PPO is on-policy and can exceed the data at the cost of exploiting the critic. Labs iterate: sample from the current policy, compare, optimise, repeat, refreshing the reward model as they go. Beacon's published relative ran six such rounds of DPO at β = 0.1 with rejection sampling in between.
bradley_terry.py: add a fifth response E that beats everything in 3 votes but loses to D once. Print the fitted scores. Then simulate a "reward hacker": a response F that the votes rate above E but whose only feature is length, and show how the fitted scale rewards it. Finally, write a function dpo_loss(logp_theta_w, logp_ref_w, logp_theta_l, logp_ref_l, beta) and reproduce the table of losses and push weights from dpo_by_hand.py.