Evaluating an application

23.1An eval is a test suite with three extra problems

You already know how to test software. Write cases, run them, count failures, block the merge if any fail. An eval keeps that shape and adds three problems that ordinary tests do not have. Each one forces a piece of machinery the rest of the chapter builds.

The last row is where most homemade evals go wrong, and it is worth stating as a rule before anything else: every number in an eval report must be traceable to a specific answer, a specific grade, and a specific reason. When a result surprises you, and it will, you need to open the row and see whether you are looking at a fact about Dispatch or a fact about your eval. In practice the most surprising results are usually bugs in the eval.

23.2Where cases come from

A case is a prompt plus the world it arrives in, plus a statement of what a passing answer does. The prompt is the easy part. The world and the statement are where the work is.

Three sources, in order of value. First, real traffic: transcripts from users, especially the ones that went wrong. A single on-call engineer's complaint that "Dispatch told me to scale to 64 workers" is worth ten invented cases, because it describes a failure that actually happened. Second, synthesis: a model can generate variations of a real case (different phrasing, different tenant, a missing detail), which is how one real incident becomes eight cases. Third, deliberate edge cases: the questions you know are hard, the questions that should be refused, the questions a careless variant would answer confidently and wrongly.

Cover both directions. If your eval only asks "does Dispatch refuse to guess when it does not know", then a Dispatch that refuses everything scores perfectly. You need the other direction too: questions that should be answered, where refusing fails. The same applies to every behaviour with a lazy extreme: tool use (call a tool when needed, and not when not), escalation, citing sources. A one-sided eval produces one-sided optimisation.

Replace live tools with fixtures. Dispatch v3 calls get_queue_depth. If the eval calls the live tool, case 7 on Monday and case 7 on Tuesday see different queues, and v2 and v3 are never tested on the same world. So each case carries a fixture: the tool readings as they were at the moment of the incident, recorded once. The eval hands the fixture to the application in place of the live call. Now every variant, every run, sees the same world, and differences between runs are differences in Dispatch.

Write expectations two people would agree on. "A good triage answer" is not an expectation; two engineers will disagree on half the answers. "Attributes the backlog to an inbound surge from one tenant; recommends scaling from 12 to 24; does not recommend more than 48 replicas" is. Each clause is a fact you can check against the runbooks. When you catch yourself writing "appropriate" or "helpful" in an expectation, replace the word with what appropriate looks like.

Don't hand over the investigation. A triage case whose prompt says "the queue is growing because tenant acme tripled its traffic; what should I do?" tests whether Dispatch can read a hint, not whether it can find the cause. Give the prompt a user would actually type ("Pager: push-queue depth > 50k") and put the evidence in the fixture, where Dispatch has to look for it.

Here is Dispatch's set. Forty cases across three behaviours, every expectation tied to a runbook from Chapter 21 or to the absence of one:

$ python code/ch23/cases.py

40 cases: {'triage': 16, 'runbook': 14, 'refuse': 10}
example: {
 "id": "triage-00-terse",
 "tag": "triage",
 "prompt": "Pager: 'push-queue depth > 50k'. Where do I start?",
 "fixture": {
  "queue_depth_trend": "growing for 12 min",
  "workers": "12/12 healthy, 0 restarts",
  "inbound_rate": "3.1x the 7-day median, 80% from tenant acme"
 },
 "expect": "attributes the backlog to an inbound surge from one tenant; recommends scaling workers from 12 to 24 (runbook 01 or 07); does not recommend more than 48 replicas; does not claim workers are failing"
}

Each case appears in two phrasings for a reason: a variant that passes the terse version and fails the 3am-panicked version is fragile in a way a single phrasing would hide. Two cases from one incident are not independent evidence, though, so when you count the effective sample size for statistics, remember you have 8 triage incidents, not 16.

23.3Grading: code where you can, a judge where you must

A grader turns an answer into a score. There are four kinds, and the rule for choosing is simple: use the cheapest one that can actually tell pass from fail on this case.

Dispatch's answers are free text, so its main grader is a judge: claude-haiku-4-5 reading the case's expectation as a rubric. The rubric is concrete (it is the expect field, clause by clause), it adds one global rule (fail any specific fact not supported by the question, the readings, or the runbooks), and it asks for a one-line verdict that starts with PASS or FAIL so the parse is deterministic. The judge sees the runbooks too, because it cannot tell a sourced fact from an invented one otherwise.

# code/ch23/run_eval.py (the judge)
RUBRIC = """You grade an on-call assistant's answer. PASS only if the answer does ALL of the following:
{expect}
Also FAIL if the answer states a specific fact (a number, name, command, credential, or cause) that is not
supported by the question, the tool readings, or the runbooks above.
Reply with exactly one line: PASS or FAIL, a colon, then a one-sentence reason."""

def judge(case, answer):
    prompt = (RUBRIC.format(expect=case["expect"]) +
              f"\n\nQUESTION: {case['prompt']}\nTOOL READINGS: {json.dumps(case['fixture'])}\nANSWER:\n{answer}")
    r = client.messages.create(model=JUDGE, max_tokens=150, system=JUDGE_SYSTEM,
                               messages=[{"role": "user", "content": prompt}])
    verdict = "".join(b.text for b in r.content if b.type == "text").strip()
    return verdict.upper().startswith("PASS"), verdict, r.usage

The judge is a different, cheaper model than the one under test. That is deliberate twice over: it keeps grading cost small relative to the thing being graded, and it avoids the judge preferring answers that sound like itself (the self-preference bias of Chapter 18 §18.5). Position bias does not arise here because each answer is graded alone, against a rubric, not against a rival answer.

Two smoke tests before you trust any grader

The null baseline. Run a constant non-answer through the whole pipeline. If the grader passes it, the grader is too lenient. Dispatch's runner has a mode for this:

$ python code/ch23/run_eval.py --null

(example output — needs an API key; this is what a healthy grader produces)
null answer  triage     0/16 pass
null answer  runbook    0/14 pass
null answer  refuse    10/10 pass

Read the last line twice. "I don't have that information; ask the on-call lead" passes every refusal case, correctly, because that is what a good refusal looks like. That is fine only because the other 30 cases fail it. An eval of refusals alone would rank the laziest possible Dispatch at 100%. The null baseline is how you see that your set covers both directions.

Calibration against humans. Take 30 to 50 answers, have a person grade them against the same expectations without seeing the judge's verdicts, then compare. The comparison is a two-by-two table, and the two numbers that matter are sensitivity (of the answers that truly pass, how many the judge passes) and specificity (of the answers that truly fail, how many the judge fails).

$ python code/ch23/calibrate_judge.py      # illustrative labels; replace with your own

12 labelled answers   agreement 75%
                judge PASS  judge FAIL
human PASS           7           1      sensitivity 88%
human FAIL           2           2      specificity 50%
true pass rate 67%  →  judge-measured pass rate 75%
disagreements to read: ['triage-03-terse', 'runbook-06-direct', 'refuse-06']

The labels in the script are invented for illustration, but the pattern is typical of a first-draft rubric: the judge rarely fails a good answer and often passes a bad one. Specificity of 50% means half of Dispatch's real failures get waved through. The fix is not statistics; it is reading the three disagreements, finding what the judge missed (say, it passed a runbook answer that promised a two-week replay the runbook forbids), and tightening the rubric until agreement on clear-cut cases is near 90%. Published work on LLM judges reports agreement with human experts in the same range as agreement between two humans, about 80%, on open-ended chat public (Zheng et al., 2023); narrow rubrics like Dispatch's should do better, and you only know yours by measuring it.

The widget shows a second, quieter effect. A judge that waves through failures inflates worse variants more than better ones, because worse variants have more failures to wave through. The gap between variants shrinks by a factor of (sensitivity + specificity − 1). With 0.92 and 0.80 that factor is 0.72: every real difference reads 28% smaller than it is. A lenient judge does not just shift your numbers; it hides your improvements.

23.4The harness: keep the plumbing out of the scores

The harness is the code between the case file and the results file. Its one job is to make sure that every score is a fact about Dispatch and nothing else. The ways it fails all have the same shape: something that is not the model ends up in the same column as the model's results.

Five rules, each one a line or two in run_eval.py:

• Call the application's real entry point. If the eval re-implements the prompt assembly, it measures a different system from the one that ships. Dispatch's runner calls the same dispatch() path with the same system prompt and runbooks.
• Errors are not failures. A rate-limit error after retries, a dropped connection, a timeout: none of these is Dispatch answering wrongly. They go to errors.jsonl. If they were scored as zero, whichever variant ran during a busy hour would look worse.
• Truncated and refused are their own statuses. An answer cut off at max_tokens is visible in the data with status: truncated; it is not counted as a wrong answer and not counted as right. A safety refusal (stop_reason: refusal, Chapter 17) is a real outcome of the system, so it is recorded, but as its own rate, not mixed into "failed the rubric".
• Assert the model that answered. The response carries the model that actually served it. If a fallback or reroute served a different model, the row measures the wrong thing; the runner sends it to the errors file.
• Measure cost and latency from the response. Tokens come from the usage block, including cache reads and writes, and are priced per model. Latency is the time from request to final token; time to first token is recorded separately because streaming users feel it more (Chapter 19). The judge's cost is recorded too, but it is the eval's cost, not the app's.

One detail links back to Chapter 20. The runbooks sit in the system prompt, byte-identical on every call, with a cache breakpoint. If the provider caches that prefix, forty cases pay for the runbooks once at full price and 39 times at a tenth of it. Whether it caches depends on the model's minimum cacheable prefix length, and the only way to know is to look: the runner prints how many calls had cache reads. If it prints zero, your prefix is below the minimum, or something in it is changing between calls.

23.5Is the difference real?

You ran v2 and v3 on the same forty cases. v2 passed 60%, v3 passed 87.5%. Is v3 better? The answer is not "27.5 points is a lot". The answer depends on how much the number would move if you ran the same variant again, and on how the forty cases split.

Pair the cases. Both variants ran on the same cases, so compare them case by case, not as two averages. Each case lands in one of four cells: both pass, both fail, only v3 passes, only v2 passes. The first two cells are ties. They say nothing about which variant is better, however many there are. The evidence is entirely in the discordant cells.

v2 wins 3 or fewer     C(17,0) + C(17,1) + C(17,2) + C(17,3)
                       = 1 + 17 + 136 + 680              =     834
all outcomes           2¹⁷                               = 131,072
one tail               834 / 131,072                     =  0.0064
two tails              × 2  (either variant wins ≥ 14)   =  0.013

The comparison script does this and a bootstrap confidence interval (resample the cases with replacement thousands of times, see how much the difference moves). Here it runs on the synthetic demo data, generated from assumed pass rates so you can use the tooling before spending anything:

$ python code/ch23/compare.py --demo       # synthetic data

40 paired cases · A=results_v2.jsonl: 60.0% · B=results_v3.jsonl: 87.5% · Δ = +27.5%  (95% CI +7.5% … +45.0%)
B won 14, A won 3, tied 23 → sign test p = 0.013
  triage   25% → 94%   (16 cases)
  runbook  71% → 86%   (14 cases)
  refuse   100% → 80%   (10 cases)
cost/case $0.0136 → $0.0176 (+29%) · p95 latency 3.59s → 4.15s
changed verdicts: ['refuse-05', 'refuse-06', 'runbook-02-context', 'runbook-02-direct', 'runbook-04-context', 'runbook-06-direct', 'triage-00-terse', 'triage-01-terse'] …

Because the data is synthetic, you know the truth behind it, and the truth is instructive. The demo generated runbook answers for both variants at the same 86% rate. The output shows runbook improving from 71% to 86%. That 15-point "improvement" is pure noise: fourteen cases, drawn twice at the same rate, happened to land differently. If you had read the per-tag table without statistics, you would have credited v3's tools with better runbook answers, and v3's tools have nothing to do with runbook questions. Small slices lie loudly.

The confidence interval says the same thing in another way: the overall improvement is somewhere between 7.5 and 45 points. Real, but very imprecise. Forty cases can tell you v3 is better; they cannot tell you how much.

How many cases you need

Since only discordant cases carry evidence, what matters is how many you expect. If two variants disagree on a fraction δ of cases, the smallest difference you can reliably detect with n cases is about 2.8 × √(δ/n) (2.8 is 1.96 for 95% confidence plus 0.84 for 80% power). Inverted: to detect a difference Δ you need about 7.8 × δ / Δ² cases.

$ python code/ch23/stats.py

n cases   minimum detectable difference (paired, 20% discordant)
    10     0.396
    40     0.198
   100     0.125
   400     0.063
  1000     0.040

to detect a 10% difference you need ≈ 157 cases
to detect a 5% difference you need ≈ 628 cases
to detect a 2% difference you need ≈ 3920 cases

v3 won 6, v2 won 2, rest tied → sign test p = 0.289
v3 won 9, v2 won 3, rest tied → sign test p = 0.146
v3 won 14, v2 won 4, rest tied → sign test p = 0.031
v3 won 30, v2 won 10, rest tied → sign test p = 0.002

The square in the denominator is the uncomfortable part. Halving the difference you want to see quadruples the cases you need. Forty cases see 20-point differences. A prompt tweak that improves Dispatch by 3 points is invisible to them, and no amount of staring at the number will make it visible. The last four lines show the same thing from the other side: a 3-to-1 win ratio is noise at 8 discordant cases, borderline at 18, and convincing at 40.

Cases and repeats are two knobs on the same dial. A model samples, so the same variant on the same case can pass one run and fail the next. Running each case three times (--reps 3) does two things: it measures that run-to-run noise directly (how often does v3 disagree with itself?), and it narrows the interval, more cheaply than writing new cases, though repeats are correlated and do not count as fully independent evidence. A useful first move for any new eval: run the same variant twice and compare it with itself. If it "beats" itself by 8 points, nothing smaller than 8 points means anything.

23.6Grading agents: outcomes and trajectories

Dispatch v3 is an agent (Chapter 22): it reads the queue, the workers, the inbound rate, a runbook, then proposes an action. Grading only the final message misses two things. An agent that reaches the right answer after twelve tool calls is a different product from one that reaches it after three: slower, more expensive, and more likely to wander somewhere it should not on the next incident. And an agent that reaches the right answer by the wrong route (guessing the cause, then calling the tools for show) will fail the first incident where the guess is wrong.

So agent evals grade two things separately:

• The outcome. Is the final answer right, against the same kind of rubric as before? For agents that change things, grade the end state rather than the narration: which actions were proposed, were any disallowed, did anything run without approval. A transcript that says "I scaled the workers" is not evidence that the workers were scaled.
• The trajectory. Tool calls made, redundant calls, calls that should have been made and were not, steps used against the budget, and any step that crossed a guardrail. These are mostly programmatic checks over the saved message list; a judge is needed only for "was this a sensible next step?".

Grade outcomes, not paths, where the path is a matter of taste. Requiring an exact tool sequence fails an agent that checked workers before queue depth, which is equally sensible. Count waste and violations; do not dictate order.

23.7The release gate

The point of all this is a decision: does the candidate ship? A release gate writes the decision down as thresholds, before you see the results, so that you cannot talk yourself into shipping the variant you already like.

A gate has several conditions and a candidate must clear all of them:

• A pass-rate floor, overall.
• No significant regression against the current version, overall.
• A floor per tag. An overall improvement that hides a collapse in one behaviour is not an improvement. The per-tag floor is what catches it.
• A cost ceiling per case, and a latency ceiling at p95, because users feel the slow tail, not the median.

Now read the demo comparison against the gate. v3 is better overall with p = 0.013, costs 29% more, and is slower at p95, all within the ceilings. But refusals fell from 100% to 80%. Two of the ten "I don't know that" questions now get a confident answer. The overall test cannot see this: two cases among forty is well inside the noise. The per-tag table can, and for refusals the stakes are asymmetric. An on-call engineer who is told the wrong failover procedure with confidence at 3am is worse off than one who is told "I don't know". So the gate holds v3 on the refusal tag. The fix is small, and it goes into the system prompt: tool readings describe the current state of Postbox; they do not answer questions about contacts, contracts, or history. Then run the suite again, both variants, and let the gate decide again.

This is the loop Chapter 24 automates: every change to the prompt, the runbooks, the tools, or the model goes through the same forty cases (then a hundred, then four hundred, as production failures become new cases), and nothing ships around the gate.

23.8Beacon's numbers, and Dispatch's

Dispatch call   system + runbooks ≈ 1,400 tokens, question + readings ≈ 150, answer ≈ 250
                input  1,550 × $5 / M   =  $0.0078
                output   250 × $25 / M  =  $0.0063
judge call      runbooks + rubric + answer ≈ 1,600 in, ≈ 40 out
                                         =  $0.0018
per case                                 ≈  $0.016      (less when the prefix caches)

v2 vs v3, 40 cases, 3 reps               = 240 case runs  ≈  $3.80
the 157 cases needed to see 10 points    × 3 reps × 2    ≈  $15