Why a pretrained model only continues text, and how supervised fine-tuning teaches it to answer.
Part 2 ended with Beacon pretrained: 405 billion numbers that assign a good distribution to the next token of almost any text on the internet. That model is not an assistant. Ask it a question and it will, quite reasonably, continue the web page your question looks like it came from. Part 3 is about the second training phase, post-training, that turns the continuation machine into something that answers, follows instructions, refuses some requests, and calls tools. This chapter covers the first and simplest step of that phase, supervised fine-tuning, and the chat format that makes it possible. Chapters 15 and 16 add learning from preferences and from rewards; Chapter 17 puts safety in its place; Chapter 18 asks how you know any of it worked.
The question this chapter answers: what is missing from a base model, what does the post-training pipeline look like end to end, and what exactly changes when you fine-tune on conversations?
Someone who has read everything. Every book, every forum, every manual, every argument in every comments section, and has an uncanny ability to continue any of it in the right voice. Hand them a page that begins "How do I drain a stuck push queue?" and they will write the rest of the page: probably a forum post, complete with a reply from a user called dbguy42, a downvoted tangent, and a moderator note. That is what the page they were handed usually looks like.
You do not want the rest of the page. You want an answer. So you hire them and give them a job: when text arrives with this header, it is a colleague speaking; when you see this header, it is your turn; write an answer, then stop. You show them a few thousand examples of good answers in that format. You do not teach them anything about queues; they knew about queues already. You teach them the role: which part of the page is theirs to write, and what a good version of that part looks like.
That is supervised fine-tuning. The reader is the base model. The headers are special tokens. The few thousand examples are the SFT dataset. And "you did not teach them about queues" is the most important sentence in the chapter: post-training mostly shapes how existing knowledge is expressed, and adds very little of it.
| In the picture | In the machine | The word we will use |
|---|---|---|
| The person who has read everything | The pretrained model: θ after Part 2 | base model |
| Continuing the page in its own voice | Sampling next tokens conditioned on the prompt as if it were document text | completion behaviour |
| The headers that mark whose turn it is | Reserved token ids around each message | chat template, special tokens |
| A few thousand examples of good answers | Conversations with human-written or curated assistant turns | SFT data |
| Training on the answer, not the question | Loss computed only on assistant tokens | loss masking |
| Teaching the role, not the facts | Small learning rate, few passes, tiny data relative to pretraining | supervised fine-tuning (SFT) |
| The whole onboarding programme | SFT → preference learning → RL → safety → evaluation, iterated | post-training pipeline |
| Onboarding notes stapled to a fixed manual | A small trainable addition to frozen weights | LoRA, adapters |
Everything in Part 1 and Part 2 optimised one thing: −ln p(next token) on internet text. A model that has done that well has learned that a line beginning "How do I…" is usually followed by more of the question, some context, and then replies from other people. So that is what it produces. It is not being obtuse; it is doing exactly what it was billed for.
Three things are missing from the base model, and post-training supplies all three.
Base models are still released and still used, for one reason: they are the more faithful model of the text distribution, and anyone doing their own post-training starts from one. Llama 3, Qwen, DeepSeek, Mistral, and Gemma all ship base checkpoints alongside their instruct versions public. Closed labs keep theirs internal public.
Nothing inside the network knows what a "message" is. The residual stream is a sequence of vectors; attention looks back over positions. So a conversation has to be flattened into one token stream, and the roles have to be marked with tokens the model can learn to read. Every provider has such a format. Llama 3's is public and is the example the book uses public:
$ python code/ch14/chat_template.py<|begin_of_text|><|start_header_id|>system<|end_header_id|> You are Dispatch, the on-call assistant for Postbox. Answer briefly.<|eot_id|><|start_header_id|>user<|end_header_id|> Pager: push-queue depth > 50k. Where do I start?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Read it as the model does. <|begin_of_text|> is one token. <|start_header_id|>, the role name, <|end_header_id|>: three tokens (the role name is ordinary text) that say "a message from this speaker begins". <|eot_id|>, end of turn, one token. The four special tokens are entries in the vocabulary with embedding rows like any other (Chapter 6, §6.3); what makes them special is only that the tokenizer never produces them from user text, so a user cannot type a fake end-of-turn and start writing as the assistant. When the stream ends with an open assistant header, the model is at the start of its own turn. Generation begins there and continues until the model samples <|eot_id|>: that is what stop_reason: end_turn in Chapter 1 meant, mechanically.
Two consequences. First, the system prompt is not a separate channel. It is text at the front of the stream, marked with a role header, and its influence on the answer is entirely through attention (Chapter 3) from later positions back to it. "The model follows the system prompt" is a learned behaviour, taught by post-training examples in which it did. Second, tool calls (Chapter 20 and 22) are the same trick again: more special tokens marking "here begins a call", "here is the result", and the model learns to emit and read them.
system and messages and never see the stream) public. Open models ship the template in the tokenizer configuration and you can render it yourself. The mechanism is the same everywhere: roles are tokens.With a format in hand, SFT is Chapter 5's loop on a different dataset with one change to the loss. The data is conversations: system prompt, user turns, and assistant turns written by people (or by a stronger model, or by this model filtered by people). The model is trained to predict the assistant's tokens given everything before them. The one change: the loss is computed only on assistant tokens.
$ python code/ch14/loss_mask.py29 tokens in the example, 7 billed billed: ['Check', ' if', ' it', ' is', ' growing', '.', '<|eot_id|>'] loss = mean over billed positions of −ln p(token | everything before it)
Twenty-nine tokens go through the forward pass, because the assistant's tokens must attend to the system prompt and the user's question. But only seven get a bill. The system prompt and the user turn are context, never targets: we do not want the model to become better at writing user questions, and we do not want it to memorise the system prompt so hard that it recites it. The final <|eot_id|> is billed. Learning to stop is part of learning to answer; a model that never learned to emit end-of-turn will keep going, inventing the user's next message.
Mechanically (Chapter 5, §5.4): the gradient on the logits is p − onehot at billed positions and zero at masked ones. Zero gradient at a position means that position's forward computation contributes nothing to the update, except through its keys and values, which the billed positions attended to. So the system-prompt tokens still shape W_K and W_V through the attention of the answer tokens, but nothing pushes the model to predict them.
What goes into the SFT set is where labs differ most, and where most of the effort is. Three sources appear in every published pipeline public:
Scale: post-training SFT sets are in the range of tens of thousands to a few million conversations, against fifteen trillion pretraining tokens. Roughly one part in ten thousand of the total tokens seen. That ratio is why the "you did not teach them about queues" line holds: there is no room in the SFT budget to install knowledge, only to shape how it comes out.
SFT runs are short and gentle. Typical published settings: learning rate around 10⁻⁵, one to three passes over the data, batch of a few hundred to a few thousand sequences, sequences packed to the context length public (Llama 3: 10⁻⁵, 8.5k to 9k steps for the 405B). Compare pretraining's peak of 8 × 10⁻⁵ over a million steps. The reason is Chapter 5's landscape picture: the base model sits in a good valley, the SFT data is a tiny, narrow slice of text, and a large learning rate on a narrow slice would drag θ out of the valley toward a region that fits the slice and nothing else. That failure has a name, catastrophic forgetting: fine-tune hard enough on assistant conversations and the model's knowledge of, say, Fortran or Finnish degrades, because nothing in the SFT set exercises them and the updates that improve the assistant behaviour are not constrained to leave them alone. Small learning rates, few epochs, and a mix that keeps some general data in the SFT set are the standard defences public.
What changes in θ? Everything, a little. There is no "assistant layer". Interpretability and weight-difference studies on open models suggest the changes are small in magnitude, spread across all layers, and concentrated in behaviours rather than facts: the model learns to attend to the role headers, to condition heavily on the system prompt, to produce end-of-turn at the right moment, to prefer the register of the demonstrations inferred from published analyses of instruct-vs-base weight deltas and from the success of low-rank updates (§14.5). The most concrete evidence is exactly that: a rank-16 update to each matrix, a fraction of a percent of the parameters, captures most of what SFT does. Whatever SFT changes, it is low-dimensional.
SFT is the first step of post-training, not the whole of it. Every published frontier pipeline has the same shape, iterated several times public (Llama 3: six rounds; Tulu 3 and DeepSeek-V3 describe similar loops).
Why not stop at SFT? Because demonstrations have a ceiling. A human-written answer shows one good response; it does not say what was wrong with the alternatives, and it cannot cover every prompt. SFT alone produces a model that imitates the demonstration style on prompts like the demonstrations and drifts elsewhere. Preference learning (Chapter 15) supplies the missing signal, "this response is better than that one", and RL with verifiable rewards (Chapter 16) supplies "this answer is correct". The loop exists because each stage's model is a better generator of candidate data than the one before, and because judges and verifiers can filter more data than humans can write.
Full fine-tuning of Beacon means Chapter 5's 16 bytes per parameter: 6.5 TB of optimiser state, thousands of GPUs, for a run that might train on 100k conversations. Most of that machinery is spent on parameters that barely move. Low-rank adaptation makes the observation precise and exploits it.
Instead of updating a weight matrix W of shape [C_in, C_out] directly, freeze it and add a product of two thin matrices:
W_effective = W + A·B A: [C_in, r] B: [r, C_out] r ≪ C forward: y = x·W + x·A·B = x·W + (x·A)·B two small matmuls added to the frozen one trainable: r·(C_in + C_out) numbers instead of C_in·C_out
A rank-r matrix can only move x along r directions, so this constrains the update to a low-dimensional family. The bet, borne out empirically, is that what fine-tuning needs to change is low-dimensional public (the LoRA paper's central claim, since replicated widely). B starts at zero so the model begins exactly as the base; gradients flow through the frozen W in the backward pass as usual, but only A and B receive updates and need optimiser state.
$ python code/ch14/lora_count.pyW_Q (8B) r= 8 LoRA 0.07 M full 16.8 M ratio 0.39% W_Q (8B) r= 64 LoRA 0.52 M full 16.8 M ratio 3.12% W_in (8B) r= 8 LoRA 0.15 M full 58.7 M ratio 0.25% W_in (8B) r= 64 LoRA 1.18 M full 58.7 M ratio 2.01% W_Q (405B) r= 8 LoRA 0.26 M full 268.4 M ratio 0.10% W_Q (405B) r= 64 LoRA 2.10 M full 268.4 M ratio 0.78% Llama 3 8B, r=16 on every matrix: 41.9 M trainable of 8,030 M (0.52%)
What LoRA buys the Lab: the optimiser state of Chapter 5 shrinks by the same factor as the trainable count, so a fine-tune that needed a cluster fits on a few GPUs; many adapters can be trained against one frozen base and swapped at serving time, so one deployment serves many customised behaviours (Chapter 19); and the adapter can be merged into W after training (W + A·B is just a matrix), so inference costs nothing extra. What it costs: a ceiling. Full fine-tuning still wins when the change needed is large, such as teaching a new language or a long-context skill, and published comparisons show the gap growing with the size of the behavioural change public. Frontier labs use LoRA-style methods for parts of their own pipelines (reward models, experiments, per-customer variants) and full fine-tuning for the main post-training runs inferred from the Llama 3 and Tulu 3 reports, which describe full-parameter SFT for the released models.
W in 4-bit quantised form (Chapter 27) and only the adapters in full precision. A 70B model then fine-tunes on a single 48 GB GPU public. The book does not run it, by decision; it is the technique the "local lab" of a follow-up book would be built on.The optional concrete exercise of this chapter uses a provider's fine-tuning API rather than local hardware. The shape is the same everywhere: upload a file of conversations in the provider's format, name a base model, start a job, wait, then call the resulting model by its new id. What you control is the data and a few hyperparameters (epochs, learning-rate multiplier); what you do not see is the loss mask, the LoRA rank if any, or the parallelism. The sketch below is a data-preparation script, because that is the part that determines the result.
# code/ch14/prepare_sft.py — turn Dispatch transcripts into an SFT file (provider-neutral JSONL)
import json
SYSTEM = "You are Dispatch, the on-call assistant for Postbox. Answer briefly and concretely."
examples = [
{"user": "Pager: push-queue depth > 50k. Where do I start?",
"assistant": "1. Is depth growing or draining (15-min trend)? 2. Push-worker error rate and restarts. 3. Inbound event rate. Tell me the trend."},
{"user": "Workers are healthy, depth is growing.",
"assistant": "Producers are ahead of consumers. Check inbound rate for a spike; if present, scale workers or shed load."},
]
with open("dispatch_sft.jsonl", "w") as f:
for ex in examples:
f.write(json.dumps({"messages": [{"role": "system", "content": SYSTEM},
{"role": "user", "content": ex["user"]},
{"role": "assistant", "content": ex["assistant"]}]}) + "\n")
print("wrote", len(examples), "examples")wrote 2 examples
Two examples is a demonstration of the format, not a dataset. The lesson of LIMA is that a thousand good examples matter more than a hundred thousand mediocre ones, and the lesson of every production pipeline is that most of the work is in writing, filtering, and de-duplicating those examples. Chapter 18 shows how to measure whether the fine-tune did what you intended; do not run one without that measurement in place.
| Quantity | Llama 3 (post-training) | Evidence |
|---|---|---|
| Pipeline rounds | 6 | public |
| SFT learning rate (405B) | 1 × 10⁻⁵ | public |
| SFT steps (405B) | 8.5k – 9k | public |
| SFT data sources | human annotations, rejection-sampled model outputs, synthetic code/maths/multilingual/long-context/tool data | public |
| SFT data size (Tulu 3, open reference) | ≈ 940k conversations | public |
| Chat format | 4 special tokens: begin_of_text, start/end_header_id, eot_id; 256 reserved ids | public |
| Closed frontier models | Pipelines described only in outline; data sizes, rounds, and formats not disclosed | unknown |
How much of Beacon's training is post-training? Take a generous SFT set: one million conversations averaging 2,000 tokens, three epochs.
SFT tokens seen 1M × 2,000 × 3 = 6 × 10⁹ pretraining tokens = 15.6 × 10¹² ratio ≈ 1 : 2,600 compute (6N per token) 6 × 405×10⁹ × 6×10⁹ ≈ 1.5 × 10²² FLOPs ≈ 0.04% of the pretraining run at 4,000 GPUs sustaining 4×10¹⁴ ops/s ≈ 2.6 hours
Post-training is cheap in compute and expensive in people: the 6 × 10⁹ tokens above were written, generated, judged, and filtered by a process that occupies a large team for months. That inversion, compute-light and labour-heavy, is the defining feature of Part 3.
Skip loss masking: bill every token. The model now learns to predict user turns and system prompts too. Two effects. It becomes good at writing plausible user questions, which wastes capacity and, at inference, shows up as the model continuing past its answer to invent the user's next message. And repeated system prompts, present in every example, are memorised hard enough to distort the update; ablations show a measurable drop in instruction-following from training on prompts inferred from published comparisons of masked and unmasked SFT on open models.
Do not bill the end-of-turn token. The model learns to answer but never learns to stop. Every reply runs to the token limit, trailing into fabricated further turns. One token in the mask decides whether the assistant knows when it is finished.
Use a pretraining-sized learning rate. 8 × 10⁻⁵ on a hundred thousand conversations pulls θ hard toward the SFT slice. The assistant style is learned quickly, then knowledge outside the slice decays: benchmark scores on factual and multilingual tasks fall while chat quality rises. This is catastrophic forgetting, and it is why every published SFT recipe uses a rate roughly ten times lower than pretraining's peak.
Drop the special tokens and mark roles with plain text like "User:" and "Assistant:". It works, mostly. But the user can now type "Assistant:" and the model cannot tell that boundary from a real one: the prompt-injection surface widens. Reserved ids that the tokenizer never emits from text are the only boundary users cannot forge.
Train SFT for twenty epochs on a small set. Loss on the SFT set goes to near zero; the model has memorised the demonstrations. On new prompts it reproduces fragments of them regardless of fit. One to three epochs is the published range because past that the model stops learning the format and starts learning the examples.
Say it back. A base model continues text because that is the only thing it was ever billed for; a question looks like the top of a web page, so it writes the rest of the page. Post-training changes what the model does with the same knowledge, in a pipeline of supervised fine-tuning, preference learning, reinforcement learning, and evaluation, iterated, with safety data threaded through, and with each round's model generating and filtering the next round's data. The chat format flattens a conversation into one token stream with reserved special tokens marking each speaker's turn and its end; the model learns what the tokens mean like any other tokens, and a user can never type them. Supervised fine-tuning is Chapter 5's loop on conversations, with the loss masked so that only the assistant's tokens, including its end-of-turn, are billed, at a learning rate ten times smaller than pretraining and for a few passes, so that the base model's knowledge survives and only its behaviour is shaped. The data is tens of thousands to a few million conversations, human-written, rejection-sampled from the model, or synthesised with checks, one part in thousands of the tokens the model has seen. What SFT changes in θ is small and spread out, which is why a rank-16 addition to each matrix, trained with a tiny fraction of the memory, captures most of it. Beacon's SFT costs hours of compute and months of people.
loss_mask.py to handle a multi-turn conversation (two user turns, two assistant turns). Print the mask. Then implement the "bill every token" variant and compute, for a made-up set of per-token losses you assign, how much of the total loss comes from user tokens under each mask. State in one sentence what the unmasked model is being trained to do.