What fifteen trillion tokens are, where they come from, and what was thrown away to get them.
Chapter 8 said how much data a run of Beacon's size needs: on the order of fifteen trillion tokens, several times the compute-optimal amount, because inference cost rewards over-training. This chapter is about those tokens. Chapter 5 showed that given the architecture, the data and the loss are the entire input to what the model becomes; the loss is fixed, so the data is where a lab's judgement lives. The map position: Part 2, the corpus, before hardware and parallelism, because the corpus is decided before the run and cannot be changed during it.
The question this chapter answers: how does a lab turn the raw internet into a training set, what does each stage remove and why, and how does it know when it is running out?
A city's water supply. The intake sits on a river that carries everything: rain, runoff, silt, sewage from upstream, and the occasional dead cow. Nobody drinks from the intake. The water passes through screens that catch the large debris, then settling tanks, then sand filters, then a chemical stage that pulls out what the filters cannot see. What comes out is a small fraction of what went in, and it is the only part anyone will drink.
Three things about the plant matter for what follows. The engineers do not inspect each litre; they set rules and let the volume flow through. Water that recirculates gets counted only once, because the reservoir's capacity is measured in distinct litres, not in litres passed. And the plant blends several sources, a river, a reservoir, boreholes, in proportions chosen for the city's needs rather than the sources' sizes: the borehole water is scarce and clean, so it is used sparingly and never wasted.
The river is the web crawl. The screens and filters are the pipeline. Recirculating water is duplication. The blend is the data mixture. And the question the city planner asks every year, "is there enough water for the population we are growing into?", is the question this chapter ends on.
| At the plant | In the machine | The word we will use |
|---|---|---|
| The river at the intake | A web crawl: petabytes of HTML, most of it useless | raw crawl (Common Crawl and private crawls) |
| Screens for large debris | URL blocklists, HTML-to-text extraction, language identification | coarse filtering |
| Settling tanks and sand filters | Rules on length, repetition, symbol ratio, boilerplate | heuristic quality filters |
| Counting recirculated water once | Removing exact and near-duplicate documents | deduplication (MinHash) |
| The chemical stage that sees what filters cannot | A trained classifier scoring each document's usefulness | model-based quality filtering |
| Testing for a specific contaminant | Removing text that overlaps evaluation benchmarks | decontamination |
| Blending sources in chosen proportions | The fraction of each batch drawn from each source | data mixture, mixture weights |
| Using scarce clean water more than once | Sampling a small high-quality source more than one pass | epochs per source, upsampling |
| Water made at a desalination plant | Text generated by a model rather than written by a person | synthetic data |
Every published frontier corpus is dominated by the web, with code, reference works, and multilingual text added deliberately. The sizes below are what the open efforts report; closed labs do not publish theirs, and the consensus assumption is that they are of similar composition at similar or larger scale inferred.
| Source or dataset | What it is | Size | Evidence |
|---|---|---|---|
| Common Crawl | A non-profit's monthly crawl of the public web since 2008; the raw material for nearly every open corpus | ≈ 100 snapshots, each hundreds of TB of HTML | public |
| FineWeb | 96 Common Crawl snapshots filtered and deduplicated per snapshot | 15 T tokens | public |
| FineWeb-Edu | FineWeb filtered again by a classifier trained on an LLM's "educational value" scores | 1.3 T (strict) / 5.4 T (lenient) | public |
| DCLM-baseline | Common Crawl filtered by a fastText classifier trained on instruction-style positives | ≈ 4 T tokens | public |
| Dolma (OLMo) | Web, code, papers, books, Wikipedia, with a fully documented pipeline | 3 T tokens | public |
| RedPajama-v2 | 84 snapshots with quality signals attached rather than applied | ≈ 30 T tokens | public |
| The Pile | 22 curated sources; the 2020 reference corpus | 825 GiB (≈ 300 B tokens) | public |
| Code | Permissively licensed repositories (The Stack and successors), deduplicated by file and by repository | ≈ 1 T tokens usable | inferred from published Stack sizes |
| Llama 3 pretraining corpus | Meta's own crawl plus curated sources; pipeline described, data not released | 15.6 T tokens | public |
The Llama 3 report gives the mixture it settled on: roughly 50% general knowledge, 25% mathematical and reasoning text, 17% code, and 8% multilingual public. Read those numbers against the row above them. Code and mathematics are a quarter of the tokens the model sees, far above their share of the web, because the Lab chose to weight them up. That choice is the subject of §9.5.
Raw HTML is not text, and the text inside it is mostly not worth training on: navigation menus, cookie banners, machine-translated spam, the same press release on four hundred sites. A pipeline is a sequence of cheap-to-expensive stages, each keeping a fraction of what it receives. The stages below are those Llama 3, FineWeb, and Dolma describe; the order and thresholds differ between labs but the shape is universal public.
Extraction. The text of a web page is a small fraction of its HTML, and the boilerplate around it (menus, footers, "related articles") repeats across millions of pages. Extractors such as trafilatura pull the main content; Llama 3 built its own parser and reports that extraction quality mattered more than expected, in particular preserving the structure of mathematics and code rather than flattening it public.
Language identification. A fastText classifier scores each document's language; documents below a confidence threshold (FineWeb: 0.65 for English) are dropped public. For a multilingual corpus the same step routes documents into per-language streams with their own downstream thresholds.
Heuristic filters. Rules that are cheap to compute and catch the obvious: documents that are too short or too long, whose mean word length is absurd, that are mostly symbols or digits, that contain the same line many times, that lack the common stopwords every real paragraph has, that consist mostly of bullet points or ellipses. The Gopher paper's rule set and C4's are the two reference lists, and FineWeb adds its own from inspecting what survived public. Llama 3 adds a filter that compares a document's token distribution to the corpus average and drops outliers, which removes a class of garbage no single rule names public.
The web repeats itself. The same news article is syndicated to hundreds of domains; the same product description appears on every reseller; the same license text heads a million source files. If the corpus keeps every copy, the model trains on the repeated text many times over while seeing everything else once, and the repeated text is not the valuable part. Published ablations show that deduplicated corpora train better models at equal token count, reduce verbatim memorisation, and make held-out evaluation honest, because a "held-out" document with ten copies in training is not held out public (Lee et al., 2022).
Three granularities, all used together:
Turn each document into its set of shingles: every run of n consecutive words (FineWeb uses n = 5). The Jaccard similarity of two documents is
J(A, B) = |A ∩ B| / |A ∪ B| shared shingles ÷ all shingles in either
Computing this for every pair of a billion documents is impossible. MinHash replaces each set by a short signature: for each of k hash functions, hash every shingle and keep the minimum. The key fact: for one hash function, the probability that two sets have the same minimum equals their Jaccard similarity, because the minimum lands on some element of the union, and it is the same element for both sets exactly when that element is in the intersection. So the fraction of matching signature positions estimates J, with error shrinking as k grows. FineWeb uses k = 112, arranged as 14 bands of 8, so that only pairs matching in a whole band are ever compared: that makes the search linear instead of quadratic and targets pairs with J above about 0.75 public.
A: the detective looked at the body and knew the killer was still in the room B: the detective looked at the body and knew that the killer was still in the room 3-word shingles: A has 13, B has 14 (one inserted word creates two new shingles and breaks one) shared: 11 union: 16 Jaccard = 11 / 16 = 0.688 MinHash with k = 16 hash functions: 10 of 16 minima agree → estimate 0.625 MinHash with k = 128: 79 of 128 agree → estimate 0.617
$ python code/ch09/minhash_by_hand.pya vs b: shingles 13 vs 14, shared 11, union 16 exact Jaccard = 0.688 MinHash estimate with k= 16: 0.625 MinHash estimate with k=128: 0.617 a vs c: shingles 13 vs 10, shared 0, union 23 exact Jaccard = 0.000 MinHash estimate with k= 16: 0.000 MinHash estimate with k=128: 0.000
The estimate is noisy at k = 16 and still a little low at 128 (this particular hash draw), but both land near 0.65, and either would flag A and B as near-duplicates at a 0.6 threshold. Document C, about Postbox, shares no shingle and scores zero. This is the whole computation; the engineering is doing it for ten billion documents.
One published surprise is worth carrying. FineWeb's authors first deduplicated globally across all 96 snapshots and found the model trained on the result was worse than one trained on per-snapshot deduplication public. Their explanation: global dedup removed everything that appeared in more than one crawl, which is disproportionately the stable, high-quality pages, and kept the churn. Deduplication is not "more is better"; it is a threshold decision with a quality cost on both sides.
Heuristics catch garbage. They do not distinguish a well-written recipe from a well-written proof, and a lab that wants its model to reason wants more of the second. The step that does this is a trained classifier, and three published recipes show the range:
Notice the loop. A model is used to grade the data for the next model. That is now standard, and it is the first place in the book where a frontier model's behaviour is shaped by a previous model's opinions rather than by raw human text. Chapter 14 and 16 make that loop explicit and much larger.
Quality filtering is a trade of quantity for quality, and the trade is not free. FineWeb-Edu's strict cut keeps 9% of the tokens. For a small model that is a good trade; for a model that needs 15 T tokens it may leave too little, and the lab must either accept a lenient threshold, repeat the strict slice, or blend. That is §9.5.
The training set is not one pile. It is a set of sources, each with a mixture weight: the fraction of every batch drawn from it. Weights are chosen, not inherited from source size, and choosing them is one of the highest-leverage decisions in the run.
Epochs. If a source of 0.4 T tokens supplies 25% of a 15.6 T run, it is seen about ten times. Repeating data is not useless, but its value decays: the "data-constrained scaling" experiments found that up to about four repetitions cost little compared with fresh data, and past that the return falls off quickly public (Muennighoff et al., 2023). A mixture that implies forty epochs of one source is spending compute to memorise it.
How weights are chosen. Not by intuition alone. Llama 3 reports running scaling-law experiments on small models with candidate mixtures and picking the mixture whose small-model losses on target tasks extrapolated best public. This is Chapter 8's machinery pointed at the data instead of the model size: the same predictable loss curves that say how big to build say what to feed.
Curriculum. The mixture is not constant. Two changes late in the run are now standard public:
T² makes long sequences expensive and most documents are short. Llama 3 extended the context from 8k to 128k in six stages over the final 800 B tokens, mixing in long documents so the model learns to use the room public.Contamination. The benchmarks used to evaluate models (Chapter 18) are text, and text is on the web. A model that has seen a benchmark's questions and answers during pretraining scores well on it for the wrong reason. Decontamination removes documents that overlap the benchmarks: Llama 3 flags any document sharing an 8-token span with a benchmark item, reports scores on both the full and the "clean" subsets, and treats a large gap as a warning public. The volume removed is tiny; the effect on trustworthy numbers is not. It is imperfect by construction, since paraphrases and translations of test items escape n-gram matching, which is one reason Chapter 18 spends time on evaluations that cannot be memorised.
Synthetic data. Text written by a model rather than by a person. It enters the corpus in three ways, each with a different justification:
The hazard is documented under the name model collapse: a model trained on the outputs of models trained on model outputs loses the tails of the distribution generation by generation, until the rare and the surprising are gone public (Shumailov et al., 2024). Labs manage this by keeping synthetic data a minority of pretraining, anchoring it to verifiable signals (does the code run, does the answer match), and reserving the heaviest use for post-training, where the targets are narrower (Chapters 14 to 16).
The city planner's question. Chapter 8 established the demand: a compute-optimal run wants roughly twenty tokens per parameter, and over-trained models use two to four times that. The supply is the stock of human text that can be crawled, cleaned, and used. The most cited estimate puts the stock of indexed, public, human-written text at around three hundred trillion tokens, and projects that frontier runs at current growth rates exhaust the usable fraction sometime between 2026 and 2032 public (Villalobos et al., Epoch, 2024). After quality filtering the usable fraction is a small multiple of what today's largest runs already consume.
Three responses are visible in published work, and Beacon's successors will use all of them: repeating the best data a few times (which the epoch experiments say is nearly free up to four); synthetic data anchored to verification; and other modalities, since image, audio, and video carry information that text does not and are far from exhausted (Chapter 25).
| Quantity | Llama 3 (405B) | Evidence |
|---|---|---|
| Training tokens | 15.6 T | public |
| Mixture | ≈ 50% general knowledge · 25% math and reasoning · 17% code · 8% multilingual | public |
| Deduplication | URL, document (MinHash), and line level; lines seen > 6× per 30 M-document bucket removed | public |
| Quality signals | heuristics, token-distribution outlier filter, fastText and DistilRoBERTa classifiers distilled from Llama 2 judgements | public |
| Decontamination | 8-token overlap with benchmark items; clean-subset scores reported | public |
| Curriculum | 8k context for most of the run; 128k reached in six stages over the last 800 B tokens; annealing on high-quality math and code | public |
| Closed frontier models | Corpus sizes, sources, and mixtures not disclosed; same pipeline shape assumed | unknown / inferred |
How many times does Beacon see its best data? Take the Lab's mixture and plausible pool sizes.
run length 15.6 T tokens math & reasoning weight 25% → 3.9 T tokens drawn math & reasoning pool say 0.4 T after filtering (the web holds little of it) epochs of that pool 3.9 / 0.4 ≈ 10 × well past the ≈4× where repetition stops paying code weight 17% → 2.65 T drawn; pool ≈ 1.5 T → ≈ 1.8 × fine web weight 50% → 7.8 T drawn; pool ≈ 12 T → ≈ 0.65 × most of the web is never seen
The arithmetic exposes the pressure. The sources the Lab most wants are the scarcest, so they are repeated most, and the repetition has diminishing returns. That is why the same report describes generating mathematical and code data synthetically, and why the annealing stage concentrates the scarce data where it counts most, in the final steps. The pool sizes here are guesses inferred; the weights and the run length are published, and the conclusion holds for any plausible pools.
Skip deduplication. The corpus contains ten copies of every syndicated article and a thousand of every license header. Those texts are effectively trained on for ten to a thousand epochs while everything else gets one; the model memorises them verbatim, regurgitates them under light prompting, and its held-out loss is flattering because the held-out set is contaminated by copies. Lee et al. measured all three effects public.
Skip quality filtering and train on all of the crawl. Most tokens are boilerplate, spam, and machine-generated filler. The model learns to predict boilerplate very well, which is worthless, and the useful text is a small minority of the gradient signal. DCLM's ablations show a filtered 4 T beating an unfiltered corpus several times larger at equal compute public.
Train only on the cleanest source. Wikipedia and textbooks alone: a few hundred billion tokens. Repeated fifty times, the model memorises them and never sees dialogue, code, informal register, or the long tail of facts that only the messy web contains. Quality without breadth produces a narrow model.
Deduplicate globally and aggressively. Everything that appears in more than one crawl is removed, and stable, canonical pages appear in every crawl. FineWeb's measured result: a worse model than per-snapshot dedup public. Thresholds are choices with costs on both sides.
Skip decontamination. Benchmark scores rise without the model getting better; the Lab believes its model reasons when it recalls. Decisions about architecture, data, and release are then made on false evidence. The tokens involved are a rounding error; the damage is to every downstream judgement.
Replace the web with model-generated text. The first generation is fluent and clean. Each successive generation trained on the previous one's output loses the rare, the odd, and the specific, until the model writes average text about average things. Model collapse, measured public.
Say it back. The corpus is the only input to training besides the loss, and it is built, not found. Raw crawls are extracted to text, tagged by language, and passed through cheap heuristic filters that remove obvious garbage. Duplicates are removed at three levels, with near-duplicates found by MinHash: shingle each document, keep the minimum of each of k hashes, and treat matching signatures as an estimate of Jaccard overlap; too little dedup lets repeated text dominate, too much removes the stable pages. A trained classifier then scores what remains for usefulness, and the classifier is increasingly a distilled opinion of an earlier model. Benchmark overlaps are removed so that evaluation can be trusted. The survivors are organised as sources with mixture weights chosen by small-model experiments, not by size; scarce sources are repeated, with diminishing returns past a few epochs; the mixture shifts late in the run toward long documents and toward the highest-quality data as the learning rate decays. Synthetic text fills gaps and supplies supervision the web never had, kept in check because models trained on model output lose the tails. And the supply is finite: a few hundred trillion tokens of public text, a fraction of it usable, against runs that already consume fifteen trillion. Beacon's corpus is that pipeline's output, weighted toward mathematics and code far beyond their share of the web.
minhash_by_hand.py to a banded index: split the 128-position signature into 16 bands of 8, hash each band, and report two documents as candidates if any band matches. Generate 2,000 short documents by perturbing 100 originals, and measure how many true near-duplicate pairs the index finds against brute-force Jaccard, and how many pairs it compared.