What happens on the other side of the API: batching, caches, speculation, quantisation, and the price of a token.
Part 4 turns the map around. Parts 2 and 3 followed Beacon from raw text to a finished assistant; from here on you stand where Dispatch stands, outside the API, and ask what is on the other side of it. This chapter is the machinery that answers a request: how thousands of conversations share one copy of the weights, why the first token and the rest arrive at different speeds, and how a rental price for a GPU becomes a price per million tokens. Chapter 7 gave you the two facts this chapter builds on: prefill is compute-bound, decode is memory-bound, and the KV cache is the memory that decode needs.
The question this chapter answers: how is a frontier model served to millions of users at once, and what determines what a token costs and how fast it arrives?
A bus route through a city. The bus is enormous, costs the same to run whether it carries one passenger or four hundred, and takes exactly one minute per stop regardless of load. Every passenger wants to go a different distance: some three stops, some forty. The route only makes money if the bus is full, so the operator's whole job is keeping seats occupied.
The naive operator waits at the depot until forty passengers have gathered, then drives them all until the last one gets off, with seats emptying stop by stop and nobody allowed to board mid-route. The good operator lets people on at every stop, the moment a seat frees, and never waits at the depot at all. Same bus, same fuel, four times the fares.
Three more tricks make the route pay. Passengers who share the first part of a journey sit in the same seat until their paths diverge. A scout on a bicycle rides ahead and guesses where the next few passengers want to go, so the driver checks several stops at once instead of one at a time. And the bus itself is rebuilt lighter, with thinner seats, so it burns less fuel per stop and fits more people.
The bus is a decode step: reading all the weights once. The passengers are sequences in the batch. Letting people on at every stop is continuous batching. Sharing a seat is the prefix cache. The scout is speculative decoding. The lighter bus is quantisation. And the fare is what this chapter derives at the end.
| On the bus route | In the machine | The word we will use |
|---|---|---|
| One stop, one minute, any load | One decode step: stream every weight through the GPU once, for every sequence in the batch | decode step |
| The passengers on board | The sequences being generated at the same time | batch |
| Boarding at any stop as seats free | Admit a new request into the batch at any step; retire finished ones immediately | continuous batching |
| Each passenger's seat | That sequence's KV cache, held in GPU memory for its whole ride | KV cache, allocated in pages |
| Sharing a seat for the common part of a journey | Reuse the KV cache of an identical prompt prefix across requests | prefix caching (prompt caching) |
| The scout guessing the next stops | A small model drafts several tokens; the big model verifies them in one pass | speculative decoding |
| The lighter bus | Weights and cache stored in fewer bits | quantisation |
| How long until the bus first moves | Time from request to first output token | TTFT |
| Minutes per stop as felt by one passenger | Time between successive output tokens of one stream | TPOT |
| Fares per hour for the operator | Tokens per second across all users | throughput |
Chapter 7 left you with the asymmetry. Prefill pushes a whole prompt through the model in one pass: thousands of tokens, each multiplied by every matrix, arithmetic that saturates the GPU. Decode pushes one token per sequence per step: the same weights read from memory, almost no arithmetic per byte read. Put numbers on it for Beacon-class serving.
405B parameters, weights in 1 byte (fp8), 8 GPUs of 80 GB at 3.35 TB/s each (H100 spec sheet, public) prefill, 4,000-token prompt 2N × 4,000 = 3.2 × 10¹⁵ ops ≈ 0.8 s of 8 GPUs at 50% utilisation decode, one step, ONE sequence read 405 GB / (8 × 3.35 TB/s) ≈ 15 ms, doing 0.8 × 10¹² ops (≈ 0.2 ms worth) decode, one step, 64 sequences read 405 GB once, 64 tokens out ≈ 15 ms, same as for one
The decode line is the whole chapter. Producing 64 tokens costs the same 15 ms as producing 1, because the time is spent reading weights, not multiplying them. Everything below is a way of filling that step with useful work, or of reading less.
Two limits stop the batch from growing forever. The first is arithmetic: at some batch size the multiplies finally take longer than the read, and the step becomes compute-bound like prefill. For the numbers above that crossover is in the hundreds of sequences. The second, and in practice the binding one, is memory: every sequence in the batch needs its KV cache resident, and Chapter 7's calculator put that at hundreds of kilobytes per token. §19.3 returns to it.
The obvious way to batch is to collect requests, run them together, and return when they all finish. It wastes most of the bus. Requests arrive at random times and generate replies of different lengths, so a static batch spends much of its life waiting for its longest member, with finished sequences occupying slots that could hold new ones, and new arrivals queueing for the next batch.
Continuous batching, introduced by the Orca system and now universal public, schedules at the level of the decode step rather than the request. At every step the scheduler looks at the batch: any sequence that emitted its end-of-turn token is removed, and any waiting request is admitted into the freed slot, with its prefill run at that step. The batch is a rolling population, and the step never waits for anyone.
Two consequences that you will feel from the outside. First, your request's time to first token depends on the queue: if every slot is occupied, you wait until one frees, however fast the model is. Second, mixing prefill and decode in one step means a long prompt arriving mid-batch stalls everyone else's next token for the duration of its prefill, which is why serving systems chunk long prefills into pieces spread over several steps, and why TTFT and TPOT can degrade at different times for different reasons public.
The batch is limited by cache memory, so the cache is where the second big idea lives. A sequence's KV cache grows by one token per step, and nobody knows in advance how long the reply will be. The natural allocator reserves a contiguous block sized for the maximum possible reply, for every sequence, up front. The vLLM authors measured what that costs in earlier systems: 60 to 80 percent of the allocated cache memory was reserved and never used, sitting empty while new requests were refused for lack of space public.
Their fix is the one operating systems reached fifty years ago for the same problem. Cut the cache into fixed-size blocks of, say, 16 tokens. Give each sequence a block table mapping its logical positions to physical blocks anywhere in memory. Allocate a block only when the sequence has filled its previous one. Attention over a paged cache needs a kernel that follows the table, which is PagedAttention, and the result was a two-to-four-fold throughput gain over the previous state of the art at the same latency, purely from admitting more sequences into the same memory public.
Paging also makes the third idea nearly free. If two requests share a prompt prefix, the same physical blocks can appear in both block tables. A system prompt used by ten thousand conversations is computed and stored once. This is prefix caching, exposed to you by providers as prompt caching: mark a stable prefix, and repeated requests skip its prefill entirely and pay a reduced price for the tokens they did not have to recompute. The rules you will meet in Chapter 20 (the prefix must match byte for byte; the cache lives for minutes, not days) are exactly the rules of a block table keyed by content public (per the Claude and OpenAI caching documentation).
The decode step wastes the GPU's arithmetic. Speculative decoding spends that idle arithmetic to produce more than one token per step, without changing what the model outputs public (Leviathan et al. and Chen et al., independently, 2023).
The trick has two halves. First, a small, fast draft model (a few percent of the target's size, or a cheap extra head on the target itself) generates k candidate tokens autoregressively; it is so small that k of its steps cost less than one step of the target. Second, the target model runs one forward pass over the prompt plus all k candidates at once, which is a prefill-shaped operation over k+1 positions and costs about the same as a single decode step. That pass yields the target's own distribution at every candidate position. Now compare: walk along the candidates, and at each one accept it with probability min(1, p_target / p_draft); at the first rejection, sample a corrected token from a distribution built from the two, and stop. The accepted prefix plus one correction is emitted.
The acceptance rule is chosen so that the emitted tokens have exactly the target model's distribution, as if it had decoded them itself. Nothing about quality changes; the draft only proposes, and the target has the last word on every token. If the draft agrees with the target 80% of the time per token and drafts four, the expected output is about 3.4 tokens per target step instead of 1.
draft length k = 4, per-token acceptance α = 0.8, draft step costs 5% of a target step expected tokens per target step = (1 − α^(k+1)) / (1 − α) = (1 − 0.8⁵) / 0.2 = (1 − 0.328) / 0.2 = 3.36 time per target step (relative) = 1 + k × 0.05 = 1.20 speed-up = 3.36 / 1.20 ≈ 2.8× with α = 0.6 (a weak draft): (1 − 0.6⁵)/0.4 = 2.31 tokens, ÷ 1.20 ≈ 1.9× with k = 8, α = 0.8: (1 − 0.8⁹)/0.2 = 4.33 tokens, ÷ 1.40 ≈ 3.1× longer drafts help less than better ones
$ python code/ch19/speculative_sim.pyk=2 α=0.6: 1.95 tokens/step (expected 1.96) speed-up ×1.77 k=2 α=0.8: 2.43 tokens/step (expected 2.44) speed-up ×2.21 k=2 α=0.9: 2.70 tokens/step (expected 2.71) speed-up ×2.45 k=4 α=0.6: 2.27 tokens/step (expected 2.31) speed-up ×1.89 k=4 α=0.8: 3.31 tokens/step (expected 3.36) speed-up ×2.76 k=4 α=0.9: 4.08 tokens/step (expected 4.10) speed-up ×3.40 k=8 α=0.6: 2.44 tokens/step (expected 2.47) speed-up ×1.74 k=8 α=0.8: 4.28 tokens/step (expected 4.33) speed-up ×3.05 k=8 α=0.9: 6.25 tokens/step (expected 6.13) speed-up ×4.46
The simulation matches the formula and shows the shape: acceptance rate matters more than draft length, and past a point extra draft tokens are mostly rejected and only cost time. Published results on production drafts report two- to three-fold decode speed-ups on typical text and less on unpredictable text like fresh code, where the draft guesses badly public.
Why it works at all is worth one sentence: the draft is right most of the time because most tokens are easy. "The detective looked at the" is followed by a noun a tiny model can guess; the target's job is to catch the one in five where the small model is wrong. Speculation moves the easy tokens to the cheap model and leaves the hard ones to the expensive one, and the batch-sized arithmetic that decode was wasting is what verifies them.
If the step is bound by reading the weights, read fewer bytes. Training keeps weights in 16 bits (Chapter 11); serving can store them in 8, or 4, by mapping each group of values to a small integer plus a scale factor. An 8-bit store halves the bytes per step and doubles decode throughput at the same batch; 4 bits halves it again. The cost is precision: rounding errors in the weights perturb every logit, and the question is how much.
The published picture, from open-model quantisation studies, is that 8-bit weights (int8 or the fp8 format hardware now supports natively) are essentially free in quality for large models, and 4-bit weights with good group-wise scaling lose a little on hard benchmarks and more on small models public (GPTQ, AWQ, and the fp8 results in the DeepSeek-V3 report). The KV cache can be quantised too, to 8 bits with little loss, which is why the calculator above assumed one byte per cache value. Chapter 27 goes deeper into how the mapping is chosen and what breaks; for serving, the rule is: fp8 weights and cache are the current default at the frontier, and 4-bit is a size-versus-quality choice made per deployment.
Beacon, 405B, one decode step across 8 × H100 bf16 weights 810 GB does not fit in 640 GB → 16 GPUs, or 2 nodes with tensor parallelism, ≈ 15 ms/step on 16 fp8 weights 405 GB fits, 235 GB left for cache → 8 GPUs, ≈ 15 ms/step int4 weights 203 GB fits, 437 GB left for cache → 8 GPUs, ≈ 7.5 ms/step, batch can double
The other lever on bytes is the same one Chapter 12 introduced for the cache: grouped-query and multi-head-latent attention shrink what is stored per token by 8 to 16 times, which is why those architectural choices were made with serving in mind and not only training.
A 405B model does not fit on one GPU even at fp8 with room for cache, so serving uses tensor parallelism (Chapter 11): each matrix is split across the GPUs of a node, each GPU streams its slice of the weights, and the partial results are combined over NVLink twice per block. Bandwidth adds up: eight GPUs stream eight slices at once, which is where the "8 × 3.35 TB/s" in the arithmetic above came from. Across nodes the interconnect is slower and the combine steps start to dominate, so a single replica rarely spans more than one node; scale beyond that comes from running more replicas behind a load balancer inferred from published serving-system designs.
A newer arrangement separates the two workloads onto different machines. Prefill wants many tokens per sequence and few sequences; decode wants many sequences and one token each; putting both on the same GPUs means a long prefill interrupts everyone's decode and a decode-heavy batch under-uses the arithmetic. Disaggregated serving runs prefill on one pool of GPUs, ships the resulting KV cache over the network to a decode pool, and lets each pool be tuned for its workload. DeepSeek described this split, with different parallelism strategies in each pool, for serving V3 public; the cost is moving the cache between machines, which for a long prompt is gigabytes per request.
From outside the API you see two numbers per request, and they come from different parts of the machinery.
Dispatch can measure the first two directly from the streaming API. The script times the first text event and the rest.
# code/ch19/dispatch_latency.py
import time, anthropic
client = anthropic.Anthropic()
t0 = time.perf_counter(); first = None
with client.messages.stream(model="claude-opus-5", max_tokens=600, system=SYSTEM,
messages=[{"role": "user", "content": question}]) as stream:
for _ in stream.text_stream:
if first is None: first = time.perf_counter()
response = stream.get_final_message()
t1 = time.perf_counter()
out = response.usage.output_tokens
print(f"TTFT {(first-t0)*1000:6.0f} ms")
print(f"TPOT {(t1-first)/max(1,out-1)*1000:6.1f} ms → {(out-1)/(t1-first):5.1f} tokens/s")(example output — your numbers depend on the provider's load, your prompt, and the model's thinking budget) input tokens 61 output tokens 348 TTFT 920 ms (prefill + queue + first decode step) TPOT 31.4 ms → 31.8 tokens/s for this one stream total 11.85 s in 212 stream events
Two cautions when you read such numbers. Models that think before answering (Chapter 16) spend decode steps on thinking tokens before the first visible text, so TTFT includes that work; it is not queueing. And the stream delivers text in chunks of several tokens, so per-event timing overstates TPOT; divide total time by output tokens instead, as the script does.
Everything above collapses into one derivation. Take a rental price per GPU-hour, the step time, and the batch, and you have the cost floor of an output token; take the prefill rate and you have the floor for an input token.
Beacon-class: 405B at fp8 on 8 × H100, rented at $3 per GPU-hour ($24/hour for the replica; rental prices public, 2025, vary widely) decode step 405 GB / (8 × 3.35 TB/s) = 15 ms batch 64 64 tokens per step → 4,235 tokens/s cost per second $24 / 3600 = $0.0067 cost per M output $0.0067 / 4,235 × 10⁶ ≈ $1.57 per million output tokens prefill 8 × 10¹⁵ ops/s × 50% / (2 × 405 × 10⁹) ≈ 4,900 tokens/s cost per M input $0.0067 / 4,900 × 10⁶ ≈ $1.35 per million input tokens batch 8 → $12.6 per M output; batch 256 → $0.39, but its cache needs 541 GB and only 235 GB is free
$ python code/ch19/cost_per_token.pybatch 1: step 15.1 ms decode 66 tok/s out $100.75/M in $ 1.35/M KV cache 2.1 GB batch 8: step 15.1 ms decode 529 tok/s out $ 12.59/M in $ 1.35/M KV cache 16.9 GB batch 64: step 15.1 ms decode 4235 tok/s out $ 1.57/M in $ 1.35/M KV cache 135.3 GB batch 256: step 15.1 ms decode 16940 tok/s out $ 0.39/M in $ 1.35/M KV cache 541.2 GB weights: 405 GB on 640 GB of HBM; prefill 4938 tok/s at 50% MFU
Read the batch column. At batch 1, an output token costs a hundred dollars per million; at 64 it costs a dollar and a half. The provider's economics are entirely about keeping the batch full, and the cache memory is what caps it: 256 sequences of 8k tokens need more cache than the weights left free, so the real ceiling for this configuration is around a hundred. Compare the floor with a list price: at the time of writing, the largest Claude models list at $5 to $10 per million input tokens and $25 to $50 per million output public (Anthropic pricing page, mid-2026). The gap between the floor and the price covers everything the floor ignores: idle capacity at off-peak hours, redundancy, networking, safety systems, the closed model's true size, and the cost of having trained it.
Three things follow that you can now explain rather than accept. Output tokens cost more than input tokens because they come from the memory-bound loop whose batch is capped by cache, while input tokens come from the compute-bound loop that batches almost freely. Cached input tokens are cheap because their prefill did not happen. And rate limits exist because a replica's throughput is finite and its batch is a shared resource: your tokens-per-minute cap is a seat allocation on a bus with a known number of seats.
| Quantity | Value | Evidence |
|---|---|---|
| Decode step, 405B fp8 on 8 × H100 | ≈ 15 ms | derived from spec-sheet bandwidth public |
| KV cache per token (fp8, 8 KV heads, 126 layers) | 258 KB | Chapter 7's formula on Llama 3.1 405B shapes public |
| Cache-limited batch at 8k context | ≈ 110 sequences | 235 GB free / (8,192 × 258 KB) |
| Cost floor per M output tokens at batch 64 | ≈ $1.6 | derived; rental price assumed |
| Speculative decoding gain on prose | ≈ 2–3× | public (Leviathan 2023; production reports) |
| Throughput gain from paged KV cache | 2–4× | vLLM paper public |
| Frontier list prices, input / output per M | $5–10 / $25–50 | Anthropic pricing, mid-2026 public; changes often |
| Closed providers' batch sizes, replica counts, quantisation | not disclosed | unknown |
Serve with static batching. Every batch runs until its longest reply finishes; short replies hold their slots idle; arrivals queue for the next batch. Utilisation falls by roughly half and TTFT becomes a function of the previous batch's slowest user. This was the state of the art until 2022 and the reason early API latencies were so uneven public.
Reserve contiguous cache for max_tokens. Most of the reservation is never used; the batch is capped at a fraction of what memory allows; throughput drops two- to four-fold. Setting a smaller max_tokens helps the allocator but truncates replies. Paging removes the trade-off.
Accept every drafted token. Decode becomes fast and the output becomes the draft model's. The rejection test is what makes speculation lossless; skip it and you have quietly swapped models. Some systems do offer this as a deliberate quality-for-speed setting, but it must be a setting.
Quantise a small model to 4 bits. Rounding error is a larger fraction of each weight's information when there are fewer weights to average over; 4-bit quality loss on an 8B model is measurable where on a 405B it is marginal public. Quantisation is a per-deployment decision, not a free lunch.
Put prefill and decode in one undifferentiated batch, with no chunking. A single 100k-token prompt arriving mid-batch stalls every other user's next token for the seconds its prefill takes. TPOT becomes bursty and unpredictable. Chunked prefill or disaggregation is what keeps a busy service smooth.
Say it back. A decode step streams every weight through the GPU once, at a cost that does not depend on how many sequences are in the batch, so serving is the art of keeping the batch full and the bytes few. Continuous batching admits and retires sequences at every step so no slot waits. The batch is capped by KV-cache memory, so the cache is cut into fixed blocks allocated on demand through per-sequence block tables, which also lets identical prompt prefixes share blocks and skip their prefill. Speculative decoding spends the idle arithmetic of a decode step verifying several tokens drafted by a small model, accepting them under a rule that preserves the target's distribution exactly. Quantising weights and cache to 8 or 4 bits reads fewer bytes per step and leaves more room for cache. Replicas span a node by tensor parallelism and scale by count; prefill and decode may be split onto pools tuned for each. From outside, TTFT is queue plus prefill plus a step, TPOT is a step divided by any speculative gain, and the price of a token is the replica's hourly cost divided by its throughput, which is why output costs more than input, cached input costs less than fresh, and rate limits are seats on a bus.
dispatch_latency.py five times with prompts of 50, 500, and 5,000 tokens (pad with a runbook). Plot TTFT against prompt length and TPOT against prompt length. Then send the 5,000-token prompt twice within a minute with the same system prompt and report what happens to TTFT and to the cached-token count in usage.