Scaling laws, compute-optimal training, and the arithmetic of a frontier run.
Part 1 built the machine and priced one step of it: 6N operations per token, 16 bytes per parameter. Part 2 asks how the Lab decides what to build. This chapter is the first decision and the one every other decision hangs on: how big, and trained on how much? The answer is not taste. It is a measured, published relationship between compute spent and loss achieved, and the Lab uses it to choose Beacon's size before a single GPU is switched on. Chapter 9 supplies the tokens, Chapters 10 and 11 the hardware and the parallelism, Chapter 12 the architecture, and Chapter 13 the run itself.
The question this chapter answers: why does a model with ten times the parameters and ten times the data predict better, by how much, and how does a lab turn that into a budget?
You are buying a telescope. The catalogue does not list what each one can see. It lists mirror diameter and price, and then, on the last page, one graph: the faintest star each model has resolved, plotted against mirror area. The points fall on a straight line on log paper. Doubling the area gains you a fixed fraction of a magnitude, every time, from the cheapest instrument to the observatory-grade one.
That graph changes how you shop. You no longer ask "what can this telescope see?"; you ask "how much do I want to spend, and where does the line say that puts me?". You can also read it backwards: to see a given star, you know the mirror you need, and you can decide whether to buy it, grind it, or give up. And you notice the line is straight only in these coordinates: on ordinary axes it is a curve of diminishing returns, each doubling buying less than the last, but never nothing.
Language-model loss against compute is that graph. The line is a scaling law. The catalogue was published in 2020, corrected in 2022, and every frontier lab re-derives its own version before every large run.
| In the picture | In the machine | The word we will use |
|---|---|---|
| Mirror area | Parameters N, tokens D, or their product in compute C ≈ 6ND | scale |
| Faintest star resolved | Held-out next-token loss in nats (Chapter 1) | loss L |
| The straight line on log paper | L ≈ E + A/N^α + B/D^β, a power law with a floor | scaling law |
| The floor no telescope beats | The entropy of the text itself, E | irreducible loss |
| Spending a fixed budget on the best mirror | For a given C, the N and D that minimise L | compute-optimal (Chinchilla) |
| Buying a smaller telescope and observing longer, because you will use it every night | Training a smaller N on far more than the optimal D, because serving cost is 2N per token forever | over-training |
| A star that "suddenly appears" | A benchmark score that jumps while the loss moved smoothly | emergence (and its mirage) |
Train a family of transformers of different sizes on the same data, plot the held-out loss against parameters, and you get a straight line on log-log axes. Do the same against tokens, or against total compute, and you get two more straight lines. That was the 2020 finding, and its exponents were small: loss fell roughly as N−0.076 against parameters, D−0.095 against tokens, and C−0.05 against compute public. Small exponents mean the line is shallow: a 10× increase in compute lowered loss by about 11%. But it held over seven orders of magnitude of compute, with no sign of bending, and that regularity is what made the frontier possible to plan.
Why a power law? No one has a first-principles derivation that everyone accepts, but the shape is what you would expect if the text contains features of many different frequencies, and each doubling of scale lets the model capture the next tier of rarer ones inferred. Common patterns are learned first and cheaply; each further gain requires learning something that occurs less often, which needs more data to see and more parameters to store. Diminishing returns, but at a steady rate in log-space.
The 2022 correction changed the numbers, not the shape. The Chinchilla paper trained 400 models and fitted a single equation to all of them:
L(N, D) = E + A / N^α + B / D^β fitted values (public): E = 1.69 A = 406.4 B = 410.7 α = 0.34 β = 0.28
Three terms. E is the loss you would get with infinite parameters and infinite data: the noise in the text itself. The second term is what you lose for having only N parameters; the third, for having seen only D tokens. Each shrinks as a power of its variable, so the equation is two power laws added to a floor. The exponents say how fast: since α > β, a factor of 10 in N shrinks its penalty by 100.34 ≈ 2.2×, while a factor of 10 in D shrinks its penalty by 100.28 ≈ 1.9×.
Shape check: L is a scalar in nats per token. N counts parameters (the non-embedding ones, in the original fit), D counts training tokens. The constants are specific to that paper's data, tokenizer, and architecture; every lab refits them, and the refit is the first thing a run plan contains.
Two consequences are visible in the widget. First, the curves never cross: more of either variable always helps. Second, they flatten, and they flatten toward a floor that is not zero. No amount of scale predicts text perfectly, because text is not perfectly predictable; the 1.69 is the Lab's estimate of how much genuine surprise English on the web contains. Chapter 1's perplexity gives the intuition: e1.69 ≈ 5.4 effectively equiprobable choices per token, forever.
Now fix a budget C in FLOPs. By Chapter 5's 6N rule, C = 6ND, so choosing N fixes D = C/6N. A bigger model sees fewer tokens; a smaller one sees more. Somewhere in between the loss is lowest, and the Chinchilla question was: where?
The 2020 answer had been "spend most of any increase on parameters". The 2022 answer, from the fit above and from two independent methods in the same paper, was that parameters and tokens should grow together: roughly, for every parameter, train on about 20 tokens public. The demonstration was a 70 B model trained on 1.4 T tokens that beat a 280 B model trained on 300 B tokens using the same compute public. The bigger model had been starved.
Run the fit yourself. The script grid-searches N for a range of budgets:
$ python code/ch08/chinchilla.py budget C (FLOPs) N (params) D (tokens) D/N loss
1.0e+20 6.455e+08 2.582e+10 40 2.600
1.0e+21 1.820e+09 9.156e+10 50 2.329
1.0e+22 5.168e+09 3.225e+11 62 2.139
1.0e+23 1.457e+10 1.144e+12 78 2.005
1.0e+24 4.138e+10 4.028e+12 97 1.911
3.8e+25 2.129e+11 2.975e+13 140 1.817
Llama 3 8B, 15T tokens: loss 1.949 (D/N = 1875, far past the optimum)
Chinchilla-optimal at 8B: loss 2.164 (D = 160B tokens, same N)
same compute, optimal split: loss 1.923 (N ≈ 3.55e+10 )
Notice the D/N column is not a constant 20. The published fit, taken literally, says the optimal ratio rises with budget, from about 40 at small scale to well over 100 at frontier scale public (this is the paper's "approach 3"; its other two approaches gave a flatter ratio near 20, and the disagreement is a known feature of the fits, not a typo). What every version agrees on is the direction: the 2020 practice of huge models on few tokens was wrong, and tokens matter about as much as parameters.
The Lab does exactly this before Beacon. Meta reported the procedure for Llama 3: train a sweep of small models on the intended data, fit the law, and read off the optimum for the budget in hand. Their fit predicted a 402 B model on 16.55 T tokens for 3.8 × 10²⁵ FLOPs; they built 405 B on 15.6 T public. The chapter's back-of-envelope reproduces that budget below.
The compute-optimal point minimises loss for the training budget. It ignores what happens after training. Every token the model ever serves costs 2N operations (Chapter 7), and a frontier model serves trillions of tokens over its life. A smaller model is cheaper on every one of them, forever.
So the Lab often deliberately builds a model that is too small for its training budget and trains it far past the optimum. Llama 3 8B saw 15 T tokens, about 1,875 per parameter, against a compute-optimal 160 B or so public. The fit says this cost about 0.03 nats against spending the same compute on a 35 B model. In exchange, the 8 B is four times cheaper to serve than the 35 B would be, every day, for years. Meta says this trade-off was the reason, explicitly public.
Read the widget's three curves. Training cost 6ND rises as you shrink N, because reaching the target loss with a small model takes a huge D, and eventually diverges: below some size no amount of data reaches the target, because the A/Nα term alone exceeds the gap to the floor. Serving cost 2N × lifetime tokens is a straight line rising with N. The sum's minimum depends entirely on how many tokens you expect to serve, which is why the same lab trains a 405 B near the optimum (served rarely, used to distil) and an 8 B far past it (served constantly).
The loss is a single number, and it moves smoothly. What people care about is what the model can do, and that relationship is less tidy.
Loss predicts capability, on average. Meta fitted a two-stage law for Llama 3: compute to loss on a benchmark's own text, then that loss to accuracy on the benchmark, and the second stage was a smooth sigmoid public. This is why labs trust loss curves: a lower loss on the right kind of text reliably becomes a higher score on the right kind of test.
Some scores jump. A 2022 survey collected benchmarks where models below a size scored at chance and above it scored well, with nothing in between, and called this emergence public. A 2023 rebuttal showed that most of those jumps disappear when the metric is changed: a benchmark scoring "all tokens of the answer correct" turns a smooth per-token improvement into a curve that sits at zero and then shoots up, because pk is tiny until p is near 1 public. The widget below is that argument in one picture.
The honest summary is that both are true. Loss improves smoothly and predictably; some downstream abilities track it smoothly; a few genuinely appear to turn on within a narrow range of scale, and whether those are "real" emergence or artefacts of measurement is still argued inferred. What is not argued is the planning consequence: a lab can forecast loss to two decimals before a run, and cannot forecast which benchmark will move by how much. This is why Chapter 18 spends a whole chapter on evaluation.
What scale does not buy. Scale does not change the objective. A model ten times larger is ten times better at next-token prediction on the training distribution, and that is all the law promises. It does not promise the model will follow instructions (Part 3 exists because it does not), will be truthful, will reason in a way you can audit, or will know anything that happened after its data was collected. The law is a statement about a loss on a corpus, and every consequence outside that is an empirical extra.
A fitted law is a summary of the runs that were done, and it silently carries their conditions. Four of them matter when the Lab plans Beacon.
One pass over the data. Every published fit assumes each token is seen once. Frontier runs now want more tokens than exist at the quality they need (Chapter 9), and the question of repeating data has been measured: up to about four passes over the same tokens cost almost nothing relative to fresh data, and beyond that the returns fall steeply, until at a few dozen repeats extra passes are nearly worthless public. The practical rule is that D in the law means unique tokens up to a small multiple, and a run plan that needs 30 T tokens from a 10 T corpus is quietly in the regime where the law overstates the gain.
Data quality moves the constants. Two corpora of the same size do not give the same A, B, or E. Filtering the web harder lowers the floor and steepens the curve, because the model spends less capacity on boilerplate and noise; adding code changes the optimal ratio because code is more predictable per token than prose inferred from the spread of fitted constants across published papers. This is why the constants are refit per mix and why Chapter 9 is a chapter and not a footnote.
Architecture shifts the curve, slightly. Within the transformer family the law is remarkably indifferent to detail: depth-to-width ratio, activation function, and normalisation change the loss at a given scale by amounts smaller than a 2× change in compute public. This is a liberating fact for Chapter 12: most architectural choices are second-order, and the first-order lever is always scale. The exceptions, mixture-of-experts above all, change what N even means, and Chapter 12 handles them.
The loss is measured on the training distribution. A law fitted on web text forecasts loss on web text. Loss on a specialised domain, or on a benchmark's prompts, follows its own curve with its own constants, and the two-stage fits that labs now use (compute → domain loss → benchmark accuracy) exist precisely because a single global loss does not forecast a specific skill well.
Every provider sells several models at very different prices per token, and this chapter is the reason. Serving cost is 2N per token plus the memory to hold the weights and cache (Chapter 7), so a model one-tenth the size is about one-tenth the price to run. Because over-training closes most of the loss gap at small scale, a lab can offer a small model that is far better than its parameter count once suggested, at a fraction of the cost. When Dispatch chooses a model tier in Chapter 20, the trade-off is the one in the over-training widget seen from the other side of the API: the provider already paid the training bill; you pay 2N for every token you generate, forever. The question a builder asks is not "which model is best" but "which is the smallest model that clears my quality bar", and that question is the same shape as the Lab's.
There is one more consequence worth naming because it is easy to miss. A lab that trains a 405 B model near the compute-optimal point does not primarily serve it. It uses it to make the small models better: generating synthetic training data, judging outputs, and distilling into the 8 B and 70 B (Chapters 9, 14, 27). The large model's value is realised partly through the small ones, which changes the arithmetic of whether the large run was worth doing at all. The over-training widget assumes serving cost is paid by the model that was trained; at the frontier the serving is often done by a smaller sibling.
| Quantity | Llama 3.1 405B | Evidence |
|---|---|---|
| Budget | 3.8 × 10²⁵ FLOPs | public |
| Scaling-law prediction for that budget | 402 B parameters, 16.55 T tokens | public |
| Built | 405 B parameters, 15.6 T tokens | public |
| GPUs | 16,384 H100 | public |
| Model FLOPs utilisation | 38–43% | public |
| Small models in the family | 8 B and 70 B on the same 15 T tokens, deliberately over-trained | public |
| Closed frontier models | Budgets and token counts not disclosed; the same procedure is assumed | unknown / inferred |
What did Beacon's run cost, from the published numbers alone?
FLOPs 6 × 405 × 10⁹ × 15.6 × 10¹² ≈ 3.8 × 10²⁵ sustained per GPU 989 TFLOP/s peak (bf16, dense) × 40% MFU ≈ 4.0 × 10¹⁴ FLOP/s sustained for the fleet × 16,384 GPUs ≈ 6.5 × 10¹⁸ FLOP/s wall-clock 3.8 × 10²⁵ / 6.5 × 10¹⁸ ≈ 5.9 × 10⁶ s ≈ 68 days GPU-hours 16,384 × 68 × 24 ≈ 27 million at $2.50 per GPU-hour ≈ $67 million (list-price guess; inferred) energy at ≈ 700 W per GPU, ×1.5 for the rest of the machine ≈ 28 GWh
Meta reported the pretraining spanning about two months public, consistent with the estimate; the dollar figure is a cloud list-price guess and a lab running its own cluster pays differently inferred. What matters is the structure: three published numbers and the 6ND rule pin the cost of a frontier run to within a factor of two.
Spend the whole budget on parameters, train each on 300 B tokens. This was the 2020 recipe, and it produced models like a 280 B trained on 300 B tokens. The fit says the 70 B on 1.4 T reaches a lower loss for the same compute, and the experiment confirmed it. The large model is not wrong, just starved: its B/Dβ term dominates and no amount of extra parameters reduces it.
Ignore the floor: extrapolate the straight line to zero loss. The line bends. Fit a pure power law with no E and it predicts that some finite compute reaches zero loss, and every downstream forecast made from it is too optimistic. Real fits with a floor consistently outperform those without on held-out scales public.
Fit the law on one dataset and apply it to another. The constants are properties of the data and tokenizer. A fit on English web text says nothing exact about a mix with 20% code, and the tokens-per-parameter optimum shifts. Every serious run refits on its own mix, which is why Chapter 9 comes before Chapter 13.
Believe every benchmark jump. Plan on the assumption that some ability will "turn on" at 10× scale, and you have made a decision on a curve that may be a metric artefact. Plan on loss, which is forecastable, and treat benchmark movements as things to measure after the fact.
Train the compute-optimal model and serve it. Correct for the training bill; wrong for the total bill if the model will serve trillions of tokens. The over-training widget shows the sum of the two costs, and its minimum is usually well left of the Chinchilla point for a model meant to be used.
Say it back. Held-out loss falls as a power of parameters and as a power of tokens, and the two effects add on top of a floor that is the entropy of the text: L = E + A/Nα + B/Dβ, with published constants that every lab refits on its own data. Because compute is about 6ND, a fixed budget forces a trade between size and data, and the loss-minimising split has tokens growing at least as fast as parameters, from tens to over a hundred tokens per parameter depending on the fit and the budget. That optimum ignores serving; since every served token costs 2N, models meant to be used are made smaller than optimal and trained far past it, which costs a few hundredths of a nat and saves a large multiple in serving. Loss moves smoothly and predictably; benchmark scores mostly follow it and occasionally jump, though many jumps are the metric's doing. Beacon's size was read off a refitted law for a 3.8 × 10²⁵ budget, and three public numbers plus 6ND price its run at two months on sixteen thousand GPUs.
L for (N = 10⁹, D = 2 × 10¹⁰), for (N = 10¹⁰, D = 2 × 10¹⁰), and for (N = 10⁹, D = 2 × 10¹¹). Which single 10× increase helped more, and by how many nats? Convert each loss to a perplexity and say what "effective number of choices per token" each corresponds to.C? Use the script's table to interpolate the compute-optimal N and D. Now suppose the model will serve 500 T tokens over its life: is a model half the optimal size, trained on twice the tokens, cheaper in total? Show both sums.chinchilla.py to fit the three constants E, A, B (holding the exponents fixed) to a set of synthetic (N, D, L) points you generate with noise from the published constants. How far off are the recovered constants with 20 points? With 400? Then repeat with the exponents free and observe how much less stable the fit becomes; this instability is why the paper's three approaches disagree on the optimal ratio.