Operating in production

24.1Pin everything you can

An LLM application's behaviour is the product of several artifacts, and each can change without you noticing. Write them down, on every response, so that "it got worse on Tuesday" can be answered with "Tuesday is when X changed".

Dispatch computes its prompt version as the first twelve hex digits of a hash over the system prompt and the tool definitions. Change a comma in a tool description and the version changes by itself. Every response carries it in a header and every trace records it.

# code/ch24/dispatch_service.py
PROMPT_VERSION = hashlib.sha256((agent.SYSTEM + json.dumps(agent.TOOLS, sort_keys=True)).encode()).hexdigest()[:12]

What you cannot pin is the sampling. The same request to the same model returns different text (Chapter 7 §7.3), and on current frontier models the sampling settings are set by the provider, not you public. So reproducibility for an LLM application does not mean "the same bytes out". It means "the same measured behaviour": the eval suite of Chapter 23, run against the pinned artifacts, gives the same numbers within its noise. That is the real regression test, and it is why the eval suite and the version fields are two halves of one idea.

Plan for model retirement. Providers deprecate models on a published schedule with notice public, and the successor behaves differently: different verbosity, different tool-calling habits, sometimes different API parameters. A model migration is a release like any other: run the suite on the new model, read the per-tag table, adjust the prompt, gate it, canary it.

24.2Trace every request

A trace is one structured record per request, written when the request finishes. It is the raw material for every number in this chapter, so decide its fields carefully and keep them stable. Here is the row Dispatch's service wrote for one triage:

{"ts": 1790464652, "request_id": "req_a099065236", "version": "v3.0", "prompt_version": "a7937c88f16f",
 "model": "claude-opus-5", "message_sha": "ce413675b321c289", "message_chars": 41, "status": "ok",
 "steps": 5, "tools": ["get_queue_depth", "get_worker_status", "get_inbound_rate", "search_runbooks",
 "propose_action"], "proposals": 1, "input_tokens": 1880, "output_tokens": 298,
 "cache_read_input_tokens": 4040, "cache_creation_input_tokens": 1010, "served_models": ["claude-opus-5"],
 "degraded": false, "refusal_fallback": false, "cost_usd": 0.02518, "latency_s": 0.0}

This row came from running the service against the scripted client of code/ch24/scripted_client.py, which is why its latency is zero; against the real model the same triage takes several seconds.

Every field answers a question someone will ask during an incident. status separates success from refusal, budget exhaustion, rate limiting, and errors, the same distinctions the eval harness made in Chapter 23 §23.4. served_models and degraded say whether the answer came from the model you asked for. The four token counts, priced per model, give the cost, exactly: 1,880 uncached input tokens at $5 per million, 1,010 written to the cache at 1.25 times that, 4,040 read from the cache at a tenth of it, and 298 output tokens at $25 per million come to $0.02518.

Metadata always, content rarely. Notice what the row does not contain: the engineer's message and Dispatch's answer. Those can hold customer names, device tokens, credentials pasted by mistake. The trace keeps a hash and a length, which is enough to group repeated questions and spot abuse, and is safe to keep for months and show on a dashboard. Content goes to a separate store, for a small sample of requests (Dispatch writes 5%), with restricted access and a short retention period. You need that sample: it is the only way to review what the application actually said. You do not need it for every request, and you do not want it in the same place as the metrics.

For an agent, one row per request hides where the time went. The widget below expands Dispatch's triage into its spans: five model calls, each a prefill of the growing context followed by a decode, with the tool calls between them.

Two things stand out, and both follow from Chapter 7. Decode dominates, because decode is one token per forward pass (Chapter 7 §7.2) and five calls each generate their own output. And every call re-sends the whole conversation so far: call 5 prefills everything calls 1 to 4 saw, plus their results. An agent's cost and latency grow faster than its step count, which is why the step budget of Chapter 22 is a cost control as much as a safety control.

24.3Watch the right numbers

From the traces you compute a handful of rates and distributions, per version, over a sliding window. The ones worth a dashboard for an application like Dispatch:

• Error rate, split by class. Rate limits mean you need more capacity or better pacing. Connection errors and 5xx responses are the provider's problem, but your users feel them.
• Refusal rate. A refusal returns HTTP 200 with stop_reason: refusal. It never shows up as an error. A prompt change that makes refusals triple is invisible unless you count them.
• Degraded rate: how often a request was answered by the fallback model.
• Budget exhaustion: how often the agent hit its step limit without concluding. A rise means the agent is wandering.
• Latency p50 and p95. The median is what a typical request feels like. The 95th percentile is what the unluckiest on-call engineer feels at 3am, and it moves first when something is wrong.
• Cost per request and the share of input tokens read from cache. If the cache share drops, cost jumps, and the cause is almost always something in the prompt prefix that started changing between requests.
• Tool calls per request. A slow upward creep is the agent's version of drift.

The monitor computes all of this from a trace log. The log in the repository is synthetic, six hours of generated traffic, seeded so the numbers reproduce: a stable version v3.0 on 90% of requests and a canary v3.1, a prompt change, on 10%.

$ python code/ch24/make_sample_log.py && python code/ch24/monitor.py

wrote 1200 synthetic traces to code/ch24/traces.jsonl (110 canary)
metric                        v3.0 (stable)  v3.1 (canary)
requests                               1090            110
error rate                             0.7%           0.0%
refusal rate                           1.2%           6.4%
budget exhausted                       0.7%           0.9%
degraded (fallback model)              0.6%           0.9%
latency p50 (s)                        8.19           6.76
latency p95 (s)                       14.82          13.98
cost per request ($)                 0.0140         0.0126
input tokens from cache               72.6%          73.1%
tool calls per request                 2.97           2.91
errors: {'APIConnectionError': 6, 'RateLimitError': 2}

CANARY GATE · refusals: canary 7/110, stable rate 1.2% → P(≥7 by chance) = 0.0004
  cost: pass · latency p95: pass · refusals: HOLD: do not widen the canary

Read the canary column as an operator would. v3.1 is faster and 10% cheaper, because its prompt makes answers shorter. It would look like a win on a cost dashboard. But it refuses 6.4% of requests against 1.2% for the stable version. Is that noise? If the canary refused at the stable rate, seeing 7 or more refusals in 110 requests would happen with probability 0.0004. It is not noise. The shorter prompt dropped a sentence that told Dispatch it may answer from runbooks, and now it declines runbook questions. The gate holds the canary at 10%, and the next step is to read the sampled transcripts of the refused requests.

Drift: quality falls while everything succeeds

None of the dashboard numbers measure whether Dispatch's answers are right. For that you need the same thing Chapter 23 used: a judge applying a rubric. In production you run it on the sampled transcripts. Every day, take the 5% sample, have a judge grade each answer against the general rules (cites a runbook when it uses one; does not invent numbers, names, or procedures), and plot the pass rate over time. A falling line with no deploy on the calendar means something outside your code moved: the runbooks were edited, users started asking about a new service, or the provider changed something. Sampled judge grading on live traffic is the only monitor that sees this.

24.4When things fail

An LLM API fails in a small number of distinct ways, and each deserves a different response. The mistake to avoid is treating them all as "retry, then give up", or worse, "fall back to another model".

The SDK already retries 408, 409, 429, and 5xx responses with exponential backoff and jitter, twice by default; Dispatch's service raises that to three. The widget shows why the jitter matters: twelve clients rate-limited at the same instant, retrying on the same schedule, all collide again.

Dispatch's service puts all of this in one small wrapper that the Chapter 22 agent loop uses as its client. The loop is unchanged; only the client is different.

# code/ch24/dispatch_service.py (excerpt)
class ProductionClient:
    def create(self, **kw):
        try:   # refusals are re-run server-side on the model recommended for the refusal category
            r = self.base.beta.messages.create(**kw, betas=["server-side-fallback-2026-07-01"], fallbacks="default")
        except anthropic.RateLimitError:
            raise                          # the limit is per account; another model does not help
        except anthropic.APIStatusError as e:
            if e.status_code < 500:        # our request is wrong: fail loudly
                raise
            self.degraded = True           # the SDK already retried; the provider is struggling
            r = self.base.messages.create(**{**kw, "model": DEGRADED})
        self.served.add(r.model)
        self.cost += call_cost(r.model, r.usage)
        return r

The fallbacks="default" parameter, behind its beta header, asks the provider to re-run a request that a safety classifier declined on another model chosen for that kind of refusal public (the Claude API documentation). The outage fallback is your own decision, and it deserves thought. For Dispatch, a slightly weaker answer during a provider outage is better than no answer during an incident, so the service falls back to claude-sonnet-5 and says so: the response carries degraded: true, and the trace counts it. For an application where a weaker answer is dangerous, the right fallback may be a clear error message instead.

These paths only happen during bad hours, which makes them the least tested code in most services. Test them deliberately, with a fake client that raises the error on demand:

$ python code/ch24/test_service.py

ok   test_bad_request_is_not_hidden_by_a_fallback
ok   test_happy_path_queues_but_never_executes
ok   test_outage_degrades_to_second_model_and_says_so
ok   test_rate_limit_is_passed_back_not_retried_elsewhere
ok   test_trace_never_contains_the_message
5 passed

24.5Cost is a design parameter

From Chapter 19 you know why providers price per token and why output costs several times input. From the trace you know what each request costs. What remains is knowing which levers to pull, and in which order. The first ones are free.

1. Cache the stable prefix. System prompt, tool definitions, reference documents: put them first, keep them byte-identical, mark a cache breakpoint (Chapter 20). Cached input costs a tenth of fresh input. In the synthetic log, 73% of input tokens come from the cache.
2. Keep the context lean. An agent re-sends everything on every call. A tool that returns a 5,000-token JSON blob when the model needs three numbers is paid for on every later call in the loop. Return what the model needs.
3. Cap the output. Output tokens cost five times input on current frontier pricing public (Appendix B). "Answer in at most three sentences" is a cost control.
4. Use lower effort where quality holds. Current models accept an effort setting that trades depth of reasoning for tokens. Measure with the eval suite; many routes lose nothing at lower effort.
5. Batch what is not urgent. Nightly summaries, eval runs, back-filling labels: batch APIs run asynchronously at a discount, about half price on the Claude API public.
6. Route to a smaller model, last, and only with eval evidence per route. Two models also mean two caches, so routing can cost more than it saves on cache-heavy traffic.

requests per month        300 incidents × 8 questions            ≈  2,400
cost per request          monitor.py, stable version             ≈  $0.014
model cost per month                                             ≈  $34

same traffic, cache cold  cost per request ≈ $0.026              ≈  $62
same traffic, 3× context  a tool returning full JSON logs        ≈  $90–100

24.6Rolling out a change

Every change goes through the eval suite first (Chapter 23). Passing it earns the change a place in production traffic, carefully, in stages, because forty cases cannot contain everything real users ask.

• Shadow. Send each real request to both versions. Users see only the current version's answer; the new version's answer is recorded and graded. No user risk, double the cost.
• Canary. Send a small share of requests, say 10%, to the new version, and compare its traces with the rest on the same dashboard. Widen in steps (10%, 25%, 50%, 100%) as each step passes its gate.
• A/B. A deliberate experiment: two versions at comparable shares, long enough to measure a user-facing outcome, like whether engineers accept Dispatch's proposals more often.

A canary is a statistical test, and the arithmetic of Chapter 23 §23.5 applies to it: small canaries see only large regressions, and only after enough traffic. The refusal regression above was detectable after 110 canary requests because it was large, 1.2% to 6.4%. A regression from 1.2% to 2% would need thousands.

That trade-off has no free resolution. A bigger canary exposes more users to a bad version but finds the problem sooner. Two practical answers: canary during busy hours, when traffic accumulates fastest, and pair the canary with shadow grading, so the judge sees the new version's answers on real traffic before anyone relies on them.

24.7Capstone 4: Dispatch in production

Everything in Part 4 now fits in one small service. code/ch24/dispatch_service.py wraps the Chapter 22 agent loop, unchanged, in an HTTP server with the production pieces of this chapter: computed versions, a trace per request, a sampled transcript store, typed error handling with refusal and outage fallbacks, and an approval queue. It runs against the real model with an API key, or against the scripted client with --stub, which replays one fixed triage so everything else can be exercised for free.

The approval gate changes shape in a service. In Chapter 22 the gate was an input() prompt: the loop stopped and waited for a person at the keyboard. A service cannot block an HTTP request for as long as an operator takes to decide, so propose_action now puts the proposal in a queue and tells the model, in the tool result, that it has not run and must not be described as done. The operator reads the queue and approves, and then carries out the change with their own tools. Dispatch can suggest; it cannot act.

$ python code/ch24/dispatch_service.py --stub &
$ curl -s -X POST localhost:8024/triage -d '{"message": "Pager: push-queue depth > 50k. Triage it."}'
$ curl -s localhost:8024/proposals
$ curl -s -X POST localhost:8024/proposals/P-0001/approve

Dispatch v3.0 · prompt a7937c88f16f · stub · 127.0.0.1:8024 · traces → traces.test.jsonl
{"request_id": "req_a099065236", "status": "ok",
 "answer": "Cause: an inbound surge from tenant acme (3.7x baseline); workers are healthy. I proposed scaling
            12 → 24 replicas per runbook 01; it takes about 90 s to apply. If depth is still growing 10 minutes
            after scaling, contact acme about pacing via the batch endpoint.",
 "proposals": [{"action": "Scale push workers from 12 to 24 replicas",
                "reason": "Depth 12k→58k in 5 min; workers 12/12 healthy; inbound 3.7x, 80% from tenant acme.
                           Runbook 01: healthy workers + growing queue → check inbound, scale 12→24. Runbook 07: max 48."}],
 "degraded": false, "version": "v3.0"}
[{"id": "P-0001", "action": "Scale push workers from 12 to 24 replicas", …, "status": "pending"}]
{"id": "P-0001", "action": "Scale push workers from 12 to 24 replicas", …, "status": "approved"}

The answer comes from the scripted client, so its wording is fixed. The tool calls behind it are real: the Chapter 22 loop ran them, search_runbooks found runbook 01 in Chapter 21's corpus by BM25, and the proposal went through the queue. Against the real model, the wording varies from run to run and the structure stays the same, which is exactly what the eval suite checks.

24.8The loop that improves the system

Production is where you learn what your eval suite is missing. The practice that turns that into improvement is simple and needs discipline:

1. Every complaint, every refused request that should have been answered, every judge failure in the daily sample becomes a candidate eval case, with its fixture recorded at the time.
2. A person writes the expectation, as in Chapter 23 §23.2, and the case joins the suite.
3. The next change must pass the suite, including the new case.

After six months, Dispatch's forty cases are four hundred, most of them things that actually went wrong. That is the data flywheel: usage produces failures, failures become cases, cases gate changes, better changes produce more usage. The labs run the same loop at scale when they turn user feedback into post-training data and evals (Chapter 14 and Chapter 18) inferred from their published descriptions of feedback-driven data collection. The difference for an application builder is that your loop feeds prompts, tools, and retrieval rather than weights.

Security is an operating concern too. A deployed Dispatch reads runbooks and tool results that other people can edit. Chapter 17 showed prompt injection: text in retrieved content that tries to redirect the model. In production the defences are the ones already built: tool results are marked as data, not instructions; the model's only write action is a proposal a human approves; and traces let you find the request where something strange happened. Add one monitor: the rate of proposals that operators decline. A spike means Dispatch is proposing things people reject, whether because of drift or because someone is steering it.

24.9Beacon's numbers