How anyone knows whether a model got better, and how often they are fooled. Capstone for Part 3.
Part 3 has been about changing a model's behaviour: supervised fine-tuning (Chapter 14), preferences (15), reinforcement learning (16), and safety (17). Every one of those chapters ended with the same unanswered question: did it work? Loss on held-out text (Chapter 1) is the wrong instrument for that, because post-training is not trying to lower perplexity; it is trying to make the model more helpful, more correct, more terse, or safer, and those are measured differently. This chapter is the measuring instrument. Part 4 reuses it for applications (Chapter 23), where the same statistics apply and the same traps wait.
The question this chapter answers: what does it take to say, with evidence, that model B behaves differently from model A, and why are most such claims weaker than they look?
A school district changes its maths curriculum and wants to know whether it worked. The obvious move is a test. But which test? If the new curriculum happened to include last year's exam questions as practice, scores go up and nothing was learned. If the test is multiple choice, a student who has learned to eliminate two wrong options looks like one who can do the maths. If the test is graded by teachers who know which students got the new curriculum, the grading drifts. If only twelve students took it, a three-point rise could be noise. And if the district compares its students against a neighbouring district's, the two groups had different students to begin with.
Every one of those failure modes has an exact counterpart in evaluating a language model, and each has a name: contamination, format sensitivity, judge bias, sample size, and unpaired comparison. The reason evaluation is a chapter and not a paragraph is that a frontier lab's decisions, which checkpoint to ship, which post-training recipe to keep, whether a safety intervention cost capability, are made on these numbers. Getting them wrong is expensive in both directions.
| In the picture | In the machine | The word we will use |
|---|---|---|
| The test | A task, a dataset of items, a metric, and the code that runs them | evaluation (eval); the code is the harness |
| A standardised test many schools take | A published dataset with a fixed metric that models are compared on | benchmark |
| Practising on last year's exam | Test items present in the training data | contamination |
| Guessing well on multiple choice | Scores that depend on the answer format rather than the skill | format sensitivity |
| Teachers grading their own students | A model scoring model outputs, with its own preferences | LLM-as-judge and judge bias |
| Same students, before and after | Both models answer the same prompts; compare item by item | paired evaluation |
| Twelve students is not enough | How many items are needed before a gap is not noise | sample size, significance |
| Parents voting on which school is better | Humans choose between two anonymous answers; many votes become ratings | arena, Bradley–Terry / Elo |
An eval is four things, and confusion usually comes from leaving one implicit.
task what the model is asked to do "answer this grade-school word problem"
dataset the items, with reference answers 8,500 problems, each with a numeric answer
metric how one item is scored exact match of the final number
harness the code: prompt template, few-shot "Q: … A: Let's think step by step" ×8 examples,
examples, decoding settings, parsing greedy, parse the last number after "####"
Change any of the four and the number changes. The same model on the same dataset can score 60% or 80% depending on the harness: how many examples are shown, whether it is allowed to reason before answering, how the answer is extracted public (the Llama 3 report devotes a section to reporting its harness choices per benchmark precisely because of this). A benchmark score without its harness is a number without units.
Two scoring styles matter enough to name. Log-likelihood scoring shows the model a question and each candidate answer and asks which continuation it assigns the highest probability to. No generation; cheap; usable on base models that cannot follow instructions; and it is Chapter 1's next-token game applied to whole answers. Generative scoring lets the model write an answer and parses it. Closer to real use; sensitive to formatting, verbosity, and refusals; and the only option for tasks with free-form answers. The two can disagree by tens of points on the same model, and both are legitimate; they measure different things.
Benchmarks come in families, each measuring one slice of capability. The public reports for open models list dozens; here are the ones that appear in nearly every model card, with what each actually tests.
| Family | Examples | What it measures | What it misses |
|---|---|---|---|
| Knowledge, multiple choice | MMLU (57 subjects, 4 options) public, MMLU-Pro, GPQA (graduate-level science, expert-written, hard to Google) public | Recall and recognition across domains | Anything that needs writing; guessable from elimination; saturating at the top |
| Maths | GSM8K (grade-school word problems) public, MATH (competition problems) | Multi-step reasoning with a checkable answer | Heavily contaminated; sensitive to whether the harness allows chain-of-thought |
| Code | HumanEval (164 Python functions with tests) public, MBPP, SWE-bench (real GitHub issues, run the repo's tests) public | Writing code that passes tests; SWE-bench adds navigating a codebase | HumanEval is tiny and memorised; SWE-bench depends on the agent scaffold as much as the model |
| Long context | Needle-in-a-haystack, RULER | Whether a fact placed deep in a long context can be retrieved | Retrieval is easy; reasoning over the whole context is what users want and what these barely test |
| Multilingual | MGSM, translated MMLU variants | Whether capability survives outside English | Translated tests inherit translation quality; tokenizer costs (Chapter 6) confound |
| Instruction following and chat | MT-Bench (80 multi-turn questions, judged) public, IFEval (verifiable constraints), AlpacaEval, Arena-Hard | Doing what was asked, in the way it was asked | Judged by a model; measures the judge's taste as much as the answer |
| Safety | Refusal sets, jailbreak suites, dangerous-capability evals (Chapter 17) | Whether the model declines what it should and helps with what it should | Adversaries adapt; the eval measures yesterday's attacks |
Two things to notice. Every family has a "what it misses" column, and none of them measure what a user experiences in a conversation. That is why the chat and arena rows exist, and why they are the noisiest. And the top rows saturate: when frontier models score in the high 80s or 90s on MMLU and GSM8K public (Llama 3.1 405B reports 87.3 and 96.8 respectively), the benchmark no longer separates them, and the field moves to harder ones. A benchmark has a useful life of a few years.
The training corpus (Chapter 9) is a crawl of the internet. Benchmarks are published on the internet, with their answers, and get copied into forums, tutorials, and GitHub repositories. So some fraction of any public test set is in the training data of any model trained after the test was released. The model has, in the language of Chapter 5, taken gradient steps on the answers.
Detection works by comparison against held-out data. The cleanest demonstration is GSM1k, a fresh set of grade-school problems written to match GSM8K's distribution: several open models scored up to 13 points lower on the new set than on the old public, while frontier models showed little gap. Labs decontaminate by searching the training corpus for n-gram overlap with every benchmark they report and removing or flagging matches; the Llama 3 report describes exactly this and reports "clean" scores on the unaffected subsets public. It is imperfect: paraphrases slip through, and the detection is only as good as the list of benchmarks someone thought to check.
The Lab's rule for Beacon is the one every serious lab follows: any benchmark used to make a decision is treated as contaminated until the decontamination has been run, and any benchmark used to publish a number must have its harness and contamination check published with it. Internally, the numbers that drive decisions come from private evals that were never posted anywhere.
The harness's prompt is part of the measurement. Published studies show the same model's MMLU score moving by several points when the answer options are relabelled from A/B/C/D to 1/2/3/4, when the few-shot examples are reordered, or when a trailing space is added public. Post-trained models add a second sensitivity: a chat model asked a raw multiple-choice question may refuse, hedge, or write a paragraph instead of a letter, and a harness that expects a letter scores that as wrong.
Three practices reduce the damage. Report the exact prompt. Evaluate with several prompt variants and report the spread, not just the best. And, when comparing two models, use the same harness for both, which is the paired design of §18.7. The last one is the most important: a five-point harness effect that hits both models equally cancels in the comparison; it only misleads when it hits them differently.
Most of what post-training changes has no reference answer. Is this reply more helpful? Better written? Did it follow the format? Humans can judge these, but a human judgment costs minutes and a lab wants a million of them a week. The standard substitute is a second model, given the question, one or two candidate answers, and a rubric, and asked for a verdict. This is LLM-as-judge, and it is the workhorse of every post-training pipeline described publicly public.
It works well enough to be useful: strong judges agree with human majority votes about as often as humans agree with each other, around 80% on chat-quality comparisons public (the MT-Bench study). It also has systematic biases, each of which has been measured:
Two judging designs. Pairwise: show two answers, ask which is better. Robust to the judge's absolute scale, sensitive to position, and it directly answers "did the change help?". Pointwise: show one answer, ask for a score from 1 to 10. Cheaper (one call per answer, not per pair), comparable across many models, but judges compress toward the middle of the scale and their 7 is not stable across days or prompts. Pairwise for decisions; pointwise for dashboards.
Whatever the design, the judge must be calibrated: on a few hundred items, compare its verdicts to human verdicts, and report the agreement. A judge whose agreement with humans is unknown is a random number generator with good grammar.
Human evaluation remains the reference. Labs run it in two forms. Rated evals: trained raters score outputs against a guideline, slowly, expensively, and with inter-rater agreement measured and reported public. Arenas: anonymous users type a prompt, see two answers from two unnamed models, and vote. The Chatbot Arena collects millions of such votes and turns them into a leaderboard public.
Turning votes into ratings is a statistics problem with a standard answer. Assume each model has a strength sᵢ and that model i beats model j with probability 1 / (1 + e^{sⱼ − sᵢ}). That is the Bradley–Terry model; Elo is an online approximation to it. Fit the strengths to the observed votes by maximum likelihood, anchor one model at zero, and report the rest with confidence intervals. The arena's public methodology does exactly this public.
$ python code/ch18/arena_sim.py50 votes → A: +1.06 (true +1.60) B: +2.02 (true +1.20) C: +1.49 (true +0.90) D: +0.75 (true +0.40) E: +0.00 (true +0.00) 500 votes → A: +1.61 (true +1.60) B: +1.04 (true +1.20) C: +0.94 (true +0.90) D: +0.47 (true +0.40) E: +0.00 (true +0.00) 5000 votes → A: +1.48 (true +1.60) B: +1.05 (true +1.20) C: +0.84 (true +0.90) D: +0.32 (true +0.40) E: +0.00 (true +0.00)
Arenas have their own biases: the prompts are whatever users type (short, English, chat-like), the voters are self-selected, and models can be tuned for arena appeal (long, confident, formatted) without being better at anything else. Recent arena methodology adds style controls for exactly this reason public. An arena rating is the best single number for "which model do people prefer in chat"; it is not a measurement of any specific capability.
Now the question Part 3 kept asking. You ran a post-training step. Did the behaviour change? The design that answers it has three parts, and the arithmetic of each is small enough to do by hand.
Pair everything. Both models (or the model before and after) answer the same prompts. Comparing model A on one prompt set with model B on another is comparing the prompt sets. Comparing them on the same prompts, item by item, cancels everything about the prompts, and leaves only the difference between the models. Each prompt yields one of three outcomes: A better, B better, tie.
Count decisive pairs, test the count. Ties carry no information about direction; drop them. Under the hypothesis that the change did nothing, each decisive pair is a fair coin. The exact sign test asks how surprising the observed split is under a fair coin.
Fifty prompts. After the change, the judge (position-swapped, consistent verdicts only) says: after-better on 28, before-better on 12, tie or inconsistent on 10.
decisive pairs n = 28 + 12 = 40
win rate 28 / 40 = 0.70
null hypothesis p = 0.5, each decisive pair a coin flip
two-sided p-value P(≤ 12 heads in 40 flips) × 2
= 2 × Σ_{i=0}^{12} C(40, i) / 2⁴⁰
= 2 × 0.0083
= 0.0166 ← below 0.05: the change did something
95% interval on the win rate (Wilson) [0.55, 0.82] ← wide: "70%" is really "somewhere above half"
same split with half the prompts (14 vs 6, n = 20) p = 0.1153 ← not significant
same split with double (56 vs 24, n = 80) p = 0.0005
a smaller effect at the same n (24 vs 16, n = 40) p = 0.2682 ← 60/40 on 40 pairs is noise
$ python code/ch18/sign_test.pywins 28 losses 12 ties 10 | win rate 0.70 95% CI [0.55, 0.82] two-sided p = 0.0166 wins 56 losses 24 ties 20 | win rate 0.70 95% CI [0.59, 0.79] two-sided p = 0.0005 wins 14 losses 6 ties 5 | win rate 0.70 95% CI [0.48, 0.85] two-sided p = 0.1153 wins 24 losses 16 ties 10 | win rate 0.60 95% CI [0.45, 0.74] two-sided p = 0.2682 true win rate 0.55 → about 782 decisive pairs for 80% power true win rate 0.60 → about 194 decisive pairs for 80% power true win rate 0.65 → about 85 decisive pairs for 80% power true win rate 0.70 → about 47 decisive pairs for 80% power true win rate 0.80 → about 20 decisive pairs for 80% power
The second block is the planning table. A change that wins 55% of pairs, which is a real and possibly valuable change, needs about 800 decisive pairs to detect reliably. Most internal "we tried X and it seemed better" decisions are made on far fewer, which is why they so often fail to replicate. The Lab sizes its decision evals from this table before running them.
Report the size, not just the verdict. "p < 0.05" says the coin is not fair; it does not say how bent it is. The win rate with its interval says that. And the win rate is only interpretable with the rubric it was judged on: a 70% win rate on conciseness says nothing about correctness, and post-training changes that improve one often degrade another. The Lab's release gate for Beacon is not one number but a panel: every decision eval, paired, with intervals, on the dimensions the change was meant to move and on the ones it must not.
Two different jobs. During a pretraining run (Chapter 13), evals are run on checkpoints every few thousand steps to catch divergence and to compare data mixtures; they must be fast and cheap (log-likelihood scoring on a few hundred items each) and their absolute values matter less than their trend. At release, evals are the evidence in the model card: full harnesses, decontaminated, with the prompts published, and the numbers compared against the previous model and against competitors on identical settings public. Confusing the two, by quoting a mid-training sanity check as a capability claim, is a common way to be wrong.
A model card's evidence has a hierarchy, and reading one is a skill. From strongest to weakest: paired human ratings with agreement statistics; arena ratings with intervals; paired judged evals with a calibrated judge and position swapping; decontaminated benchmark scores with the harness published; benchmark scores without the harness; and "we found it to be better". Beacon's card is expected to sit in the top three tiers for every claim it makes.
The design of §18.7, run for real, on a change you control. The behaviour: terseness. The change: a two-sentence system prompt. The claim to test: "the system prompt makes the model's answers more concise". Everything in Part 3 that changes a model's behaviour (SFT data, preference tuning, an RL reward) is evaluated the same way; the system prompt is just the cheapest change to make, so the mechanics of the eval stand alone.
arm A model, no system prompt "before" arm B same model, system prompt "Answer in at most two sentences." "after" prompts the same 40+ questions, drawn from Dispatch's domain paired judge a cheaper model, rubric on conciseness only, one character out swap every pair judged twice, positions swapped; keep consistent verdicts statistic exact sign test on decisive pairs; win rate with interval
# code/ch18/paired_eval.py (excerpt)
MODEL, JUDGE = "claude-opus-5", "claude-haiku-4-5"
TERSE = "Answer in at most two sentences. No preamble, no lists."
RUBRIC = """You are judging two answers to the same question for CONCISENESS ONLY: the better answer
is the one that is shorter while still answering the question. Ignore style and formatting.
Reply with exactly one character: A if answer A is more concise, B if answer B is, or T for a tie."""
def judge(question, a, b):
r = client.messages.create(model=JUDGE, max_tokens=5, system=RUBRIC,
messages=[{"role": "user", "content": f"Question: {question}\n\nAnswer A:\n{a}\n\nAnswer B:\n{b}"}])
verdict = "".join(bl.text for bl in r.content if bl.type == "text").strip()[:1].upper()
return verdict if verdict in "ABT" else "T"
for q in PROMPTS:
base, terse = answer(q), answer(q, TERSE)
v1 = judge(q, terse, base) # terse shown first
v2 = judge(q, base, terse) # positions swapped
# consistent "terse wins" / "base wins" / tie / judge flipped → counted as tie
...
p = 2 * sum(comb(n, i) for i in range(0, min(wins, losses) + 1)) / 2 ** n(example output — 40 prompts; your judge's verdicts and the exact counts will differ) terse wins base 142 words terse 31 words | Why does a push queue's depth grow even w terse wins base 118 words terse 28 words | Explain what a KV cache stores and why it tie base 41 words terse 34 words | What is the difference between prefill an judge flipped with position base 96 words terse 47 words | How should I triage a pager alert about e … terse wins 31, base wins 3, ties 6 (of which position-inconsistent 4) win rate 0.91 over 34 decisive pairs, exact sign test p = 0.0000
Three things to check when you run it, because they are where such evals go wrong. First, the word counts: if the "terse" arm is not actually shorter on most items, the judge's verdicts are measuring something else and the eval is broken before statistics enter. Second, the flip rate: a judge that contradicts itself on more than about a fifth of pairs is not a usable judge for this rubric, whatever its verdicts say. Third, the ties after swapping: the 40 prompts include a few whose base answer was already short, and those should tie. If none do, the judge is not reading.
Then the second half, which is the half people skip. A behaviour change is only worth shipping if it did not break something else. Run the same paired design with a correctness rubric on the same outputs. A terseness prompt that wins 91% on conciseness and loses 30% on correctness is a regression, and only the paired panel shows it. Dispatch's own release gate, built in Chapter 23, is this panel with more dimensions.
| Quantity | Llama 3.1 405B (as reported) | Evidence |
|---|---|---|
| MMLU (5-shot) / GSM8K (8-shot, CoT) / HumanEval (0-shot) | 87.3 / 96.8 / 89.0 | public |
| Decontamination | n-gram overlap against every reported benchmark; clean-subset scores reported alongside | public |
| Human evaluation | Paired comparisons against competitors across capability categories, with win/tie/loss rates and intervals | public |
| Judge model use | Reward-model and LLM-judge scoring throughout post-training data selection | public |
| Closed frontier models | Model cards report benchmark tables and some human-preference results; internal decision evals, judge calibration data, and contamination methods are mostly not disclosed | unknown |
What does a release-eval panel cost? Suppose the Lab's gate is 30 decision evals of 500 paired prompts each, judged twice (swapped) by a judge model, and each answer averages 400 output tokens with a 300-token prompt.
generation 30 evals × 500 prompts × 2 arms × 700 tokens ≈ 21 M tokens through the candidate model judging 30 × 500 × 2 orders × (300 + 2·400 + rubric 150) tokens ≈ 38 M tokens through the judge at list prices for a frontier candidate and a small judge ≈ a few hundred to a few thousand dollars per gate human rated slice 2,000 pairs × ~5 minutes × two raters ≈ 330 rater-hours
The tokens are cheap next to a training run. The rater-hours are the bottleneck, which is why judge models exist and why their calibration against the rated slice is the number that makes the rest trustworthy.
Evaluate the two models on different prompt sets. The comparison now includes the difference between the sets, which can be larger than the difference between the models. A five-point gap can appear or vanish depending on which set got the harder questions. Pairing removes this entirely; nothing else does.
Skip the position swap. With a judge that leans 0.8 toward the first slot, a true 60/40 preference reads as 77/23 in one order and 41/59 in the other. Whichever order the harness happened to use decides the verdict. The swap costs 2× judge calls and is the difference between a measurement and a coin toss with a thumb on it.
Decide on 20 prompts. A 70/30 split on 20 decisive pairs has p = 0.115. Reasonable-looking splits at this size are consistent with no effect roughly one time in eight. Repeated across a lab's hundred small decisions per month, a dozen of them are noise, and the recipe drifts on ghosts.
Use the model under test as its own judge. Self-preference bias means the "after" model rates its own style higher. Published measurements put this at several points of win rate public, enough to turn a null result into a positive one. Judge with a different family.
Trust a benchmark whose prompts are on GitHub. The next model trained on a crawl has seen them. The score measures recall of the test set plus skill, in unknown proportion. A fresh rewrite of the benchmark, or a private set, is the only way to know which.
Optimise directly on the release benchmark. Every eval becomes a training signal once it is used to select checkpoints, choose data mixtures, or tune prompts. The number goes up; the capability it was a proxy for may not. This is Goodhart's law, and Chapter 15's reward over-optimisation is the same phenomenon with a different name. Hold out evals that are never used for selection, and report those.
Say it back. An eval is a task, a dataset, a metric, and a harness, and the number it produces is meaningless without all four. Benchmarks are public evals that measure one slice of capability each, saturate within a few years, and are contaminated by the crawl that trains the next model, so labs decontaminate, publish harnesses, and keep private sets for decisions. Free-form behaviour is judged by a model against a rubric, which works at about human-level agreement but carries position, verbosity, and self-preference biases, each of which has a specific fix: swap positions, control length, use a different family, calibrate against humans. Human votes in an arena become ratings through the Bradley–Terry model, the same logistic relation between strength gap and win probability that Elo approximates. To say a change worked, pair the prompts, drop the ties, run the exact sign test on the decisive pairs, report the win rate with an interval, and size the eval from the effect you hope to see: a 55% effect needs about 800 decisive pairs, a 70% effect about 50. Then run the same design on the dimensions the change must not have hurt. Beacon's release gate is that panel, and the capstone is one row of it, run through the API on a change you made yourself.
paired_eval.py with a second rubric for correctness (judge whether each answer is factually right about the question, independently, pointwise 0/1), run both rubrics on the same outputs, and print a two-row panel: conciseness win rate with interval, correctness rate per arm with interval. Then change TERSE to "Answer in one sentence" and report what happens to both rows.