Hyperparameters, schedules, stability, and what two months of training actually look like. Capstone for Part 2.
Part 2 has assembled everything a frontier run needs: a compute budget and a target size (Chapter 8), a corpus (Chapter 9), the hardware (Chapter 10), a way to spread the model across it (Chapter 11), and an architecture (Chapter 12). This chapter is the run itself: the few dozen numbers that are chosen before it starts, what goes wrong while it happens, and how the people watching it decide what to do. It ends with the Part 2 capstone: a complete pretraining plan for a Beacon-class model, every number derived.
The question this chapter answers: once the model, data, and cluster exist, what does the Lab actually decide, monitor, and fix during a training run, and why?
A container ship leaving port for a 70-day crossing. Before departure, a few numbers are fixed that cannot be changed at sea: how much fuel, what speed profile, which route. The first hours are slow and careful, because a full-speed departure in a crowded harbour ends badly. Then a long stretch at cruising speed where, from the bridge, nothing seems to happen for weeks; the only evidence of progress is a slowly changing number on a display. Near the end, the ship slows deliberately to come in cleanly.
During the crossing, things break. A pump fails; a sensor reads garbage; a storm forces a course change. None of these is a surprise, because every crossing has them. The crew's job is not to prevent all failures but to notice them fast, have a recent position fix to return to, and keep the ship moving. The captain who has done twenty crossings knows what the display should look like on day 12, and worries only when it does not.
The fixed numbers are the hyperparameters. The careful start is warm-up. The cruise is the stable phase. The deliberate slowing is annealing. The breakages are hardware failures and loss spikes. The position fixes are checkpoints. And the display is the loss curve.
| On the crossing | In the run | The word we will use |
|---|---|---|
| Numbers fixed before departure | Learning rate, batch size, weight decay, Adam's betas, initialisation scale, clipping threshold | hyperparameters |
| Slow exit from the harbour | Learning rate ramped from near zero over the first thousands of steps | warm-up |
| Cruising speed | Peak learning rate held or slowly decayed while most tokens are consumed | stable phase |
| Slowing to come in | Learning rate decayed to zero on curated data | annealing (cool-down) |
| A pump fails | A GPU, link, or node dies; the job stops | hardware failure |
| A sensor reads garbage | Loss jumps suddenly; gradients blow up | loss spike |
| Position fix | Full state (weights, optimiser, data position) written to storage | checkpoint |
| The display on the bridge | Training loss, held-out loss, gradient norm, throughput, small benchmarks | the dashboard |
| Scaling the plan from a small vessel | Tuning on a small model and transferring the settings | hyperparameter transfer (muP) |
For all the size of a frontier run, the list of things a person chooses is short. Here is Llama 3 405B's list, which is the most complete one any lab has published public, with the reason each number is what it is.
| Hyperparameter | Llama 3 405B | Why that value |
|---|---|---|
| Optimiser | AdamW, β₁ = 0.9, β₂ = 0.95, weight decay 0.1 | Chapter 5's Adam. β₂ below the textbook 0.999 so the second-moment estimate forgets faster; a large second-moment memory is a known cause of instability at scale. Decay 0.1 keeps weights from drifting large. |
| Peak learning rate | 8 × 10⁻⁵ | Larger models take smaller steps: a 405B model's loss surface is sharper in the sense that matters, and rates that work at 8B (3 × 10⁻⁴) diverge here. The published values across sizes fall roughly as a power of N. |
| Warm-up | 8,000 steps, linear from 0 | The random-initialisation region is chaotic; full-size steps there destroy the delicate early structure (the first embeddings and heads). Thirteen hours of the run. |
| Decay | Cosine to 8 × 10⁻⁷ over 1.2 M steps | A smooth glide to one hundredth of the peak; the last stretch lets the model settle. |
| Batch size | 4M tokens → 8M → 16M (sequence 4k then 8k) | Small early, large late. The gradient of a nearly-random model is informative even from few tokens; later, the signal per token is small and the cluster needs enough work per step to stay busy (Chapter 11). |
| Gradient clipping | Global norm 1.0 | A cheap insurance policy: if a batch produces a freak gradient, it is scaled down rather than taken at full size. |
| Precision | bf16 with fp32 master weights and optimiser | Chapter 11's mixed precision. |
Two settings deserve a closer look because they are where runs most often go wrong.
Chapter 5's one-dimensional picture is the whole intuition: too small is slow, too large overshoots. A schedule spends the budget of steps as the landscape allows. Two shapes dominate current practice:
Nobody tunes the learning rate of a 405B model by trying values: one trial is the whole budget. The settings are tuned on models a thousand times smaller and transferred. That only works if the settings are expressed in a form that does not change with width, and the maximal-update parametrisation (muP) is a set of rules for initialisation scale and per-layer learning rates that makes the best learning rate approximately constant across widths public. Labs report using muP or a variant to set frontier learning rates from sweeps at a few hundred million parameters public (Cerebras, and DeepSeek's scaling-law fits for batch size and learning rate, which serve the same purpose). The practical effect is that "8 × 10⁻⁵" was not a guess; it was read off a curve fitted at small scale.
The loss curve of a healthy run is monotonic and boring. Occasionally it is not: the loss jumps by a large fraction within a few steps, and either recovers over hundreds of steps or keeps climbing until the numbers overflow. This is a loss spike, and every long run of the pre-2024 era reported them. PaLM's team saw about twenty in a run and handled each by restarting from a checkpoint roughly a hundred steps earlier and skipping the next few hundred batches public. The same data, in the same order, from a slightly different state, usually did not spike again: the spike was a coincidence of a particular batch and a particular parameter state, not a property of the data alone.
Why spikes happen is now reasonably understood, and the understanding has produced fixes that recent runs report needing rarely or never. The OLMo 2 team published the clearest account, having suffered spikes on their first run and eliminated them on the second public:
DeepSeek-V3 reported no irrecoverable loss spikes and no rollbacks across its full run public, and Llama 3's report does not mention spikes as a problem. The era in which every big run fought them appears to be over, not because the phenomenon vanished but because its causes were identified and designed out.
Between decisions, the run is watched. The people on shift look at a small number of time series, and what they look for is deviation from a shape they already know.
The Llama 3 team published their failure log, and it is the best public description of what operating a frontier run involves public. Over 54 days of pretraining on 16,000 GPUs: 466 job interruptions, 47 planned (maintenance, configuration changes) and 419 unexpected. Of the unexpected ones, about 78% were confirmed or suspected hardware, with GPU failures the largest single category, followed by host components, network, and storage. Silent data corruption, where a GPU computes wrong numbers without reporting an error, occurred and had to be caught by checking the arithmetic. Despite all this, effective training time exceeded 90%, because restarts were automatic and checkpoints were frequent.
That is the texture of a run: one interruption every three hours, around the clock, for two months, almost none of them requiring a human decision. The engineering that makes this survivable is Chapter 11's checkpointing and the automation around it; the widget below shows the trade-off that sets the checkpoint interval.
Frontier runs do not end at the cosine's tail. They end with two deliberate phases.
Annealing. During the final tens of billions of tokens, the learning rate is decayed to zero while the data mixture is shifted toward the highest-quality sources: curated code, mathematics, textbooks, and in some cases synthetic data that resembles the post-training distribution. Llama 3 annealed on its final 40M tokens with upsampled high-quality data and reported that annealing on small amounts of a benchmark's training data noticeably raised that benchmark, which they used as a cheap way to evaluate data sources public. The reason it works: with a tiny learning rate, the model makes small, precise adjustments, and the last data it sees has an outsized effect on its final state. This is also why held-out loss shows a visible drop at the end.
Long-context extension. Training at 128k tokens from the start would make attention's T² cost dominate every step (Chapter 3). So the run trains at 8k, and only at the end extends the context in stages, 8k → 16k → 32k → … → 128k, on a modest number of tokens, adjusting RoPE's base frequency at each stage (Chapter 12) and checking that short-context quality does not regress and that needle-in-a-haystack retrieval works at the new length before moving on. Llama 3 did this in six stages over about 800B tokens public.
After these the base model is done. What comes next, turning a document-continuer into an assistant, is Part 3.
| Quantity | Llama 3 405B | DeepSeek-V3 (671B MoE, 37B active) | Evidence |
|---|---|---|---|
| Tokens | 15.6 T | 14.8 T | public |
| Peak learning rate | 8 × 10⁻⁵ | 2.2 × 10⁻⁴ | public |
| Schedule | 8k warm-up, cosine over 1.2M steps | 2k warm-up, constant, then cosine decay over the last ~30%, then constant tail | public |
| Batch | 4M → 8M → 16M tokens | 3072 → 15360 sequences of 4k (≈ 12.6M → 63M tokens) | public |
| Cluster and time | 16,384 H100, ~54 days pretraining | 2,048 H800, 2.664M GPU-hours pretraining (~55 days) | public |
| Interruptions | 466 in 54 days, >90% effective time | no irrecoverable spikes, no rollbacks | public |
| Closed frontier models | Schedules, batch sizes, and failure statistics not disclosed; the same phases (warm-up, stable, anneal, context extension) are the consensus assumption | unknown / inferred | |
Everything in Part 2 in one document. The inputs are four choices; every other number is derived from them with the rules of Chapters 8 to 13. The script code/ch13/plan_the_run.py computes the plan; the widget below lets you change the inputs.
BEACON PRETRAINING PLAN (hypothetical; rules from Part 2)
1 · Inputs (chosen)
model dense, 400 B parameters, C = 16384, L = 126, 128 heads / 8 KV heads, SwiGLU, RoPE (ch 4, 12)
tokens 15 T, ≈ 37 tokens/parameter: past Chinchilla-optimal (20) on purpose, to buy
cheaper inference for the model's whole service life (ch 8)
cluster 16,384 H100-class GPUs, 989 TFLOP/s bf16 peak each; 400 Gb/s per GPU fabric (ch 10)
target MFU 40 % (Llama 3 reported 38–43 %) (ch 11)
2 · Compute and time (derived)
FLOPs 6 · N · D = 6 × 4×10¹¹ × 1.5×10¹³ = 3.6 × 10²⁵
cluster rate 16,384 × 989×10¹² × 0.40 ≈ 6.5 × 10¹⁸ FLOP/s
pure compute 3.6×10²⁵ / 6.5×10¹⁸ ≈ 5.6 × 10⁶ s ≈ 64 days
with failures one interruption per ~3 h, 30-min checkpoints, 6-min restarts → ≈ 72 days
GPU-hours 16,384 × 72 × 24 ≈ 28 M GPU-hours
3 · Batch and steps
sequence 8,192 tokens after the first phase (ch 12: extend later)
batch 4 M tokens for the first 250 B, 8 M to 3 T, then 16 M (1,953 sequences/step)
steps ≈ 62k + 340k + 750k ≈ 1.15 M
step time ≈ 5.9 s in the main phase (16 M tokens × 6 × 4×10¹¹ / 6.5×10¹⁸)
4 · Optimiser and schedule
AdamW β₁ 0.9, β₂ 0.95, weight decay 0.1, clip global norm at 1.0 (ch 5, 13)
learning rate peak 8 × 10⁻⁵ (read off small-scale sweeps under muP), linear warm-up 8,000 steps
(≈ 128 B tokens, 13 h), then WSD: hold to 90 %, decay to 0 over the last 10 %
stability QK-norm, z-loss 10⁻⁴, fixed-scale init, no decay on norm gains (ch 13)
5 · Memory and layout (per Chapter 11)
training state 16 B/param = 6.4 TB → 0.4 GB per GPU when sharded across all 16,384 (ZeRO/FSDP)
activations the real per-GPU memory user: 1,953 sequences × 8,192 tokens × 126 layers of
activations, kept only in checkpointed form (recompute in backward)
parallelism TP = 8 within a node, PP = 16 across nodes, CP = 1 (raised to 16 for the 128k stage),
DP = 16,384 / (8 × 16) = 128 replicas (Llama 3's 4D layout)
6 · Data (per Chapter 9)
mixture ≈ 50 % general web, 25 % math and reasoning, 17 % code, 8 % multilingual
epochs high-quality sources repeated up to 4×; web at most 1×; dedup at document and line level
held-out one evaluation set per source, 1 M tokens each, scored every 2,000 steps
7 · Checkpoints and monitoring
cadence every 30 minutes (≈ 300 steps): full state 6.4 TB at 2 TB/s ≈ 3 s of write
keep every 8th checkpoint permanently (scaling-law points, rollback targets)
alarms gradient norm > 3× its trailing mean; loss above its predicted curve by > 0.02 nats;
throughput < 90 % of nominal for > 10 min; any held-out set rising two evals in a row
rollback rule on a spike: restore the last checkpoint, skip 300 batches, resume; if it recurs, lower η by 20 %
8 · Final phases
annealing last 40 B tokens: η → 0, mixture shifted to curated code, math, textbooks, synthetic QA
long context 6 stages 8k → 128k on 800 B tokens, RoPE base raised each stage, needle tests and
short-context regressions gate each step
deliverable the base model: 800 GB of bf16 weights, a loss of ≈ 1.8 nats/token on held-out web,
handed to post-training (Part 3)
$ python code/ch13/plan_the_run.pycompute 6·N·D = 3.60e+25 FLOP cluster rate 16384 × 1e+15 × 0.4 = 6.48e+18 FLOP/s wall time 64.3 days (no failures) steps 937,500 (1953 sequences of 8192 per step) step time 5.9 s training state 16 B/param = 6.4 TB (0.39 GB per GPU before activations) checkpoint size 6.4 TB (full optimiser state); weights only 0.8 TB warm-up 8000 steps = 128 B tokens = 13.2 h annealing last 40 B tokens = 2,500 steps expected failures 514 over the run at one per 3.0 h lost time 7.5 days at a checkpoint every 0.5 h wall time with failures 71.8 days
The script uses a single 16M batch throughout, so its step count is lower than the plan's ramped total; everything else agrees. Compare with the published run this plan imitates: Llama 3 405B, 15.6T tokens, 3.8 × 10²⁵ FLOPs, 16,384 GPUs, 54 days of pretraining at 38–43% MFU. The plan lands within 20% of every public number, from four inputs and the rules in this Part.
Skip warm-up: start at the peak learning rate. The first steps take full-size moves in a chaotic region. The embeddings and early heads that would have formed in the first hour are scrambled; the loss either diverges outright or settles into a worse basin from which the run never fully recovers. The 13 hours of warm-up are the cheapest insurance in the plan.
Use β₂ = 0.999. Adam's per-parameter step size now remembers a thousand steps of gradient history. When a batch produces an unusually large gradient, the denominator has not caught up, the step is oversized for hundreds of steps, and a spike becomes likely. Every published frontier run uses 0.95 for this reason.
Never decay the learning rate. The model keeps bouncing around its minimum at the peak step size. Held-out loss plateaus visibly above where a decayed run ends: published comparisons put the gap at a few hundredths of a nat, which in Chapter 8's terms is the equivalent of a large fraction of the training tokens wasted. The annealing drop in the loss-curve widget is exactly what this forgoes.
Checkpoint once a day. With a failure every three hours, each one discards twelve hours of the whole cluster on average. Effective training time falls below 50%; the 64-day run takes four months. The interval in the plan is thirty minutes because that is roughly where write cost and loss-per-failure balance.
Train at 128k context from step one. Attention's cost per token grows sixteen-fold over 8k; the run slows several-fold for no gain, since the long-range skills only need a few hundred billion tokens to learn. Every published run extends context at the end.
Say it back. A run is set by a short list of numbers: an AdamW configuration with a fast-forgetting second moment, a peak learning rate read off small-scale sweeps, a warm-up of a few thousand steps, a schedule that holds or glides for most of the tokens and decays to zero at the end, a batch that starts small and grows to millions of tokens, and a clipping threshold. Stability is designed in: normalised queries and keys, a penalty that keeps logits centred, a careful initialisation, filtered data. What remains is monitored on a handful of time series, of which training loss on a log-token axis and gradient norm are the two that matter most; a spike is handled by rolling back to a checkpoint and skipping ahead. Hardware fails every few hours on a large cluster, so checkpoints are written every half hour and restarts are automatic, and the run still runs at over 90% effective time. The last tokens are annealed on the best data with the learning rate going to zero, and the context is extended in stages only at the end. From four inputs and the rules of Part 2, the plan for a 400B model on 15T tokens reproduces the published shape of a real one: 3.6 × 10²⁵ operations, 16,000 GPUs, about ten weeks.
plan_the_run.py to model the three-phase batch ramp, and to print a table of learning rate, batch size, and step time at 0%, 1%, 10%, 50%, 90%, and 100% of the tokens for both a cosine and a WSD schedule. Confirm that the total step count matches the widget.