Distillation, quantisation, pruning, and small models trained long: how frontier capability reaches cheaper machines, and how to choose among the results.
The book has followed Beacon at full size: 405 billion parameters, 810 GB of weights, served on several GPUs at a time (Chapter 19). Most tokens in the world are not generated by models that size. They come from smaller models, compressed models, and models distilled from bigger ones, because every token costs memory, bandwidth, and money. This last chapter of Part 5 explains the techniques that take capability from big to small and what survives the trip. On the map it closes the loop that Chapter 8's scaling laws opened: bigger works, and then you want it smaller.
The question this chapter answers: how do you get most of a large model's behaviour into fewer parameters or fewer bits, what is lost, and how do you choose which model to use for a job?
A master chef runs a restaurant with a long menu. She cannot open a thousand branches, so she does three things. She trains apprentices, not by handing them the menu, but by letting them watch her cook and taste every dish she makes, including the ones that went slightly wrong and why. She writes each recipe on a single card, in rounded measures: "a pinch", "a cup", instead of 3.7 grams and 241 millilitres. And she removes from the branch menu the dishes almost nobody orders, whose ingredients take up half the fridge.
The branches are not as good as the flagship. But they are nearly as good at the dishes people actually order, at a tenth of the cost, and the difference between "nearly" and "exactly" is what this chapter measures. The apprentices are distillation. The rounded cards are quantisation. The shorter menu is pruning.
| In the kitchen | In the machine | The word we will use |
|---|---|---|
| The master chef | A large, capable model | teacher |
| The apprentice | A smaller model trained to imitate it | student |
| Tasting everything she makes, not just reading the menu | Training on the teacher's full next-token distribution or its outputs | distillation (soft targets) |
| Recipes in rounded measures | Weights stored in fewer bits, with a scale per group | quantisation (int8, fp8, int4) |
| The one ingredient measured in kilos among ones measured in grams | Rare, very large values that force a coarse scale | outliers |
| Dropping rarely ordered dishes | Removing weights, neurons, heads, or whole layers | pruning |
| A small branch run for years on a narrow menu | A small model trained on far more tokens than compute-optimal | over-training (Chapter 8) |
Train a small model from scratch on text and its bill is −ln p(true next token) (Chapter 1). That bill is one-hot: it says which token came next and nothing about the others. But a trained large model knows more than the one right answer. After "the killer was still in the", it gives room 72%, house 16%, building 10%, and banana almost nothing. That spread is information: house is a reasonable alternative and banana is not. A student trained on the teacher's distribution learns that, from every single token. A student trained on the text alone has to infer it from millions of examples.
The distillation bill replaces the one-hot target with the teacher's distribution and measures the mismatch with the KL divergence (Appendix A):
loss = KL(p_teacher ‖ p_student) = Σ p_teacher(t) · ln( p_teacher(t) / p_student(t) )
It is zero only when the student's distribution equals the teacher's, and it charges the student most for putting little probability where the teacher put a lot. Hinton, Vinyals, and Dean added one knob: divide both models' logits by a temperature T above 1 before the softmax, which flattens both distributions and exposes the teacher's ranking of the unlikely tokens, the "dark knowledge" in the tail public (2015).
$ python code/ch27/distill_kl.pyT=1.0: teacher=[0.723 0.161 0.098 0.013 0.005] student=[0.791 0.065 0.065 0.039 0.039] KL(teacher‖student)=0.097 T=4.0: teacher=[0.339 0.233 0.206 0.125 0.097] student=[0.332 0.178 0.178 0.157 0.157] KL(teacher‖student)=0.027 hard-label loss on the same student: −ln p(room) = 0.234 (says nothing about house vs banana)
Five candidate next tokens: room, house, building, and two absurd ones. At T = 1 the student is too sure of room (0.79 against 0.72) and treats house and building as equally unlikely, where the teacher separates them. The KL of 0.097 charges it for both. The hard-label bill, 0.234, only says "put more on room", which would make the student more wrong about the alternatives. At T = 4 both distributions flatten, and the teacher's ranking of the tail becomes large enough to learn from. In practice the soft-target loss is multiplied by T² to keep its gradients the same size as at T = 1, and often mixed with the ordinary hard-label loss.
Distilling from outputs, not distributions. Logit distillation needs the teacher's full distribution at every position, which means running the teacher alongside the student, with the same tokenizer. When the teacher is only available through an API, there is a simpler form: have the teacher generate answers, and fine-tune the student on those answers with the ordinary SFT bill of Chapter 14. This is sequence-level distillation, and it is how most small models acquire a large model's style and skills. DeepSeek fine-tuned six smaller open models, from 1.5 to 70 billion parameters, on about 800,000 samples generated by DeepSeek-R1, and the students acquired much of its step-by-step reasoning (Chapter 16) public. The same paper reports that for small models, distilling from a strong reasoner worked better than running the reinforcement learning on the small model directly public.
Distillation is now part of how the labs build their own small models. Google trained Gemma 2's 2B and 9B models with distillation from a larger model instead of ordinary next-token targets public, and Meta's Llama 3.2 1B and 3B used logits from Llama 3.1 8B and 70B as targets after pruning public. Whether closed labs distil their small tiers from their large ones is not published, though it is widely assumed inferred from the open labs' practice. Note also a limit that matters for users of APIs: providers' terms of service commonly forbid using outputs to train competing models public.
Chapter 7 §7.2 established that decode is memory-bound: generating each token requires reading every weight once, and the arithmetic is idle most of the time. So the number of bytes per weight sets the decode speed as directly as anything can. Store weights in 16 bits and Beacon is 810 GB. Store them in 4 bits and it is about 200 GB: a quarter of the memory, four times less to read per token, on a quarter of the GPUs.
The mechanism is rounding, done carefully. Take a group of weights, say 64 or 128 consecutive values. Find a scale: the largest absolute value divided by the largest integer you can store (7 for signed 4-bit). Divide each weight by the scale, round to the nearest integer, and store the integers plus the one scale. To use a weight, multiply the integer back by the scale. Every weight is now one of sixteen levels, and the error is at most half a level.
$ python code/ch27/quantize_by_hand.pywith the outlier scale=0.1286 int4 codes=[0 0 0 7 0 0 0 0]… mean |error|=0.0325 error as % of typical weight=33% outlier removed scale=0.0154 int4 codes=[ 4 3 3 3 -1 -1 4 -1]… mean |error|=0.0042 error as % of typical weight=9%
A group of sixteen small weights, around ±0.05, with one set to 0.9. The large weight sets the scale: 0.9 ÷ 7 ≈ 0.129 per level. Every ordinary weight is smaller than half a level, so it rounds to zero. The codes read 0 0 0 7 0 0 0 0: one survivor and a group of erased weights. The mean error is a third of a typical weight's size. Take the outlier out of the group, or give it its own scale, and the scale drops eightfold, the ordinary weights get distinct levels, and the error falls to 9%.
Outliers are not a curiosity. Dettmers and colleagues found that above roughly 6.7 billion parameters, transformers develop a few feature dimensions with values far larger than the rest, in every layer, and that naive 8-bit quantisation of activations fails because of them public (LLM.int8(), 2022). Every serious method since is a way to handle them:
The KV cache can be quantised too. Chapter 7 showed it, not the weights, filling GPU memory at long context. Storing keys and values in 8 bits halves that and doubles the conversations a server holds, with small quality cost in published tests public.
What survives. The broad, repeated finding across published evaluations: 8-bit weights are close to lossless; 4-bit weights with grouping and a method like GPTQ or AWQ lose a little, more on small models than large ones, and more on hard reasoning and long outputs than on easy benchmarks; at 3 bits and below, quality falls quickly public. Those are averages. Your application's eval suite (Chapter 23) is the only measurement of whether a quantised model is good enough for your job.
Quantisation keeps every weight and stores it more coarsely. Pruning removes weights. The question is what "removing" buys on real hardware, and the answer depends on the shape of what you remove.
W_in and the matching row of W_out, Chapter 2 §2.6), attention heads, or embedding channels. The matrices get genuinely smaller and every layer after them gets cheaper. It hurts more per parameter removed, because whole detectors disappear.How do you decide what matters least? Magnitude is the simplest guess. Better guesses measure importance directly: run a calibration set through the model and record how much each neuron, head, or block actually contributes to the output, then remove the least used. Either way, pruning damages the model, and the damage is repaired by training the pruned model a little longer, ideally with distillation from the unpruned original as the teacher (§27.1). NVIDIA pruned Llama 3.1 8B to 4B parameters this way and recovered most of its quality with a distillation run of around a hundred billion tokens, a small fraction of what training a 4B model from scratch would take public (Minitron). Meta's Llama 3.2 1B and 3B were built the same way: pruned from the 8B, then distilled public.
The fourth route to a capable small model involves no compression at all: train it on far more data. Chapter 8 showed that the compute-optimal recipe gives roughly 20 training tokens per parameter, and that this optimum ignores inference. A model served for years to millions of users is cheaper overall if it is smaller and trained much longer than compute-optimal: more training compute once, less inference compute forever.
tokens per parameter compute-optimal (Chinchilla) ≈ 20 Llama 3 8B 15 T tokens ÷ 8 B parameters ≈ 1,875 public SmolLM2 1.7B 11 T tokens ÷ 1.7 B parameters ≈ 6,500 public
Loss keeps falling well past the compute-optimal point, slowly (Chapter 8's curves flatten but do not stop), and every bit of it is paid once and enjoyed on every token served. The other lever is the data itself. Microsoft's Phi models showed that small models trained on carefully filtered and synthetic "textbook-quality" data can match much larger models trained on raw web text on reasoning and code benchmarks public (Gunasekar et al., 2023). Chapter 9's filtering pipeline, pushed hard, is a compression technique too: it removes the tokens that teach little.
In practice the routes combine. A current small open model is often a larger model pruned, then distilled from a strong teacher on a very long, heavily filtered corpus, then post-trained as in Part 3, then shipped in 16-bit and 4-bit versions.
All of this produces a ladder of models at every price. The question for an application builder is which rung to stand on. The procedure is short, and it is Chapter 23's discipline applied to the choice:
For Dispatch the answer is clear from its own numbers. It serves a few thousand requests a month (Chapter 24 §24.5), so the model bill is tens of dollars at the largest tier, and a wrong triage at 3am costs far more than the difference. The largest model is the right choice, and nothing in this chapter changes that. The calculation flips at consumer scale. A million requests a day at a cent each is ten thousand dollars a day, and then a model that is a tenth of the price and clears the floor is worth weeks of engineering.
| Quantity | Value | Evidence |
|---|---|---|
Distillation with temperature, T² scaling | soft targets carry the teacher's ranking | Hinton et al. 2015 public |
| R1 distilled students | 1.5 B – 70 B, ≈ 800 k samples | DeepSeek-R1 report public |
| Outlier features in activations | emerge ≈ 6.7 B parameters | Dettmers et al. 2022 public |
| Llama 3.2 1B, 3B | pruned from 3.1 8B, distilled from 8B and 70B logits | Meta public |
| Llama 3 8B tokens per parameter | ≈ 1,875 | 15 T ÷ 8 B public |
| Closed labs' small tiers: distilled, pruned, quantised? | not disclosed | unknown |
What does quantisation do to serving Beacon? Weights only, on 80 GB GPUs, ignoring the KV cache.
format bits per weight weights GPUs for weights alone bf16 16 810 GB 11 int8 / fp8 8 405 GB 6 int4, groups of 128, 4 + 16/128 = 4.125 209 GB 3 one 16-bit scale per group decode time per token ≈ bytes read ÷ total memory bandwidth (Chapter 7): bf16 on 11 GPUs: 810 GB ÷ (11 × 3.35 TB/s) ≈ 22 ms int4 on 3 GPUs: 209 GB ÷ ( 3 × 3.35 TB/s) ≈ 21 ms
The per-token speed is about the same, because the bf16 version spreads its larger weights over more GPUs. The difference is the hardware: three GPUs instead of eleven for the same speed, so roughly a quarter of the cost per token, with more room left for the KV cache that Chapter 7 showed dominates at long context. That is why 4-bit weight-only quantisation is standard for serving open models, and why the question "what does it cost in quality?" is worth an afternoon with your eval suite.
Distil with hard labels only. Train the student on the teacher's top token at every position. It learns the teacher's answers and none of its uncertainty: house and banana look equally wrong. It needs far more data to reach the same place, and it is overconfident where the teacher hedged. Sequence-level distillation from sampled answers keeps some of the spread; logit distillation keeps all of it.
Quantise to 4 bits with one scale per matrix. One outlier per matrix sets the scale, and most ordinary weights round to zero, as in the by-hand example. The model's output degrades sharply. Per-group scales cost an eighth of a bit per weight and are what makes 4-bit usable.
Quantise activations to 8 bits naively in a large model. The outlier feature dimensions that appear above a few billion parameters are clipped or crush everything else in their tensor. LLM.int8() documented the collapse; the fixes all keep those dimensions in higher precision or rescale around them.
Prune half the weights, unstructured, and expect twice the speed. The matrices are the same shape and the GPU does the same multiplications. Nothing is faster unless the zeros follow a pattern the hardware can skip. Structured pruning is what saves time.
Prune and ship without repair. Removing heads or blocks shifts everything downstream of them. Without a short training run, ideally distilling from the original, the pruned model's loss is much worse than its size would suggest. The repair run is part of the method.
Pick a model from a leaderboard. The model that ranks third in general can be first on your task, or ninth. Only your eval suite measures your task.
Say it back. Serving cost is set by bytes of weights read per token and by the number of GPUs those bytes need, so making models smaller is worth a great deal. Distillation trains a small student on a large teacher's full next-token distribution, measured by KL divergence and softened by a temperature, so every token carries the teacher's ranking of the alternatives; when only outputs are available, fine-tuning the student on the teacher's answers is sequence-level distillation, and it is how small models inherit reasoning styles. Quantisation stores weights in 8 or 4 bits with a scale per small group; rare, very large outlier values force coarse scales, so methods keep groups small, keep activations in higher precision, compensate rounding error, or rescale important weights. Pruning removes weights; only structured removal of neurons, heads, or whole blocks saves time, the residual stream is what makes removing blocks survivable, and a short distillation run repairs the damage. Small models also get strong by being trained far past compute-optimal on filtered data. The routes combine, and the result is a ladder of models at every price. Choose a rung with your own eval suite: set a quality floor, test several rungs, and pick the cheapest that clears it at your volume. For Dispatch that is still the largest model; at consumer scale it rarely is.
[0.12, −0.05, 0.31, 0.02, −0.40, 0.07] to signed 4-bit (levels −7 to +7): compute the scale, the integer codes, the dequantised values, and the mean absolute error. Then replace −0.40 with −2.0 and repeat. How many of the other five weights round to zero?distill_kl.py: make the student's logits trainable and take 200 gradient steps minimising T² · KL at T = 4, then report the student's distribution at T = 1. Repeat minimising only the hard-label loss on "room". Compare the two students' probabilities for "house" and "building".