Where the labs depart from the plain transformer, and what each departure buys.
Part 1 built the plain transformer: dense MLPs, full multi-head attention, one context length. Chapters 8 to 11 showed what it costs to train one at frontier scale, and that the cost has three faces: parameters (memory), operations per token (compute), and the T² of attention (both). Every architecture change the frontier labs have made in the last three years attacks one of those three faces. This chapter is the catalogue of those changes, with the mechanism and the trade-off for each, and the evidence from the labs that publish. Chapter 13 then puts one of these architectures through a training run.
The question this chapter answers: given the plain transformer, what do frontier models change, why does each change help, and what does it cost?
A hospital. The naive design gives every patient every specialist: the cardiologist, the dermatologist, and the neurologist all examine every person who walks in. Thorough, and absurd. The real design has a triage nurse who reads the chart and sends each patient to two or three relevant specialists. The hospital employs forty specialists but each patient sees three, so the hospital can afford forty.
The same hospital keeps records. The naive design copies every patient's full history to every department. The real design keeps one compact summary per patient and lets each department expand it as needed. And the hospital has a rule about consultations: most doctors only consult colleagues on the same floor, which is fast, while a few senior doctors can call anyone in the building.
The triage nurse is a mixture-of-experts router. The compact summary is a compressed key-value cache. The floor rule is sliding-window attention with a few global layers. None of these change what the hospital does; they change what it costs to do it.
| In the picture | In the machine | The word we will use |
|---|---|---|
| Forty specialists, three per patient | The MLP replaced by E expert MLPs; each token uses k of them | mixture of experts (MoE); total vs active parameters |
| The triage nurse | A small learned matrix that scores every expert for a token and keeps the top k | router, top-k routing |
| No specialist overbooked | A loss term or bias that spreads tokens across experts | load balancing |
| One compact record per patient | Fewer key/value heads than query heads, or a compressed latent per token | GQA, MQA, MLA |
| Consult only your floor | Each token attends only to the previous w tokens | sliding-window attention |
| A few senior doctors who can call anyone | Some layers keep full attention | local-global interleaving |
| Extending the building without retraining the staff | Making RoPE work past the trained length | context extension (position interpolation, base scaling, YaRN) |
From Part 1 and Chapters 8 to 11, per token through a dense model of N parameters at context T:
compute ≈ 2N (forward) + attention's 2·T·C per layer
memory weights: 2N bytes to serve, 16N to train (Ch 5)
KV cache: 2 · L · (KV heads) · d · 2 bytes per token, per conversation (Ch 7)
attention scores are T² per head per layer; FlashAttention hides the memory, not the arithmetic (Ch 10)
Chapter 8 said the loss falls with N and with training tokens D. So a lab wants N large. But the compute per token and the serving memory both scale with N, and a large N that must be read on every decode step (Chapter 7) is exactly what makes inference expensive. The first architecture change breaks the link between "parameters that exist" and "parameters that run".
Recall the MLP of Chapter 2: expand, bend, compress, with the hidden width holding thousands of learned detectors, and recall from Chapter 4 that the MLPs hold 70% of a dense model's parameters. Now replace the single MLP in each block with E separate MLPs, the experts, each with its own W_in and W_out. Add a tiny matrix, the router, that maps the token's vector to E scores. Softmax the scores, keep the top k, and compute the block's MLP output as the weighted sum of just those k experts' outputs. The other E − k experts do nothing for this token.
dense block: x += MLP(norm x) one MLP, always
MoE block: s = softmax(norm(x) · W_router) [E] scores
top-k indices e₁ … e_k, weights g₁ … g_k (renormalised)
x += Σᵢ gᵢ · MLP_{eᵢ}(norm x) k of E MLPs, chosen per token
The parameter count now splits in two. Total parameters: attention plus all E experts, what must be stored. Active parameters: attention plus k experts, what is multiplied for each token, which by the 2N rule of Chapter 4 is what sets compute per token. Mixtral 8×7B has 8 experts, uses 2, and has 46.7 B total but 12.9 B active parameters public. DeepSeek-V3 has 256 routed experts plus one shared expert per layer, uses 8 routed, and has 671 B total, 37 B active public. The 405 B dense Llama runs every parameter for every token; DeepSeek-V3 stores 1.7× as many parameters and runs one eleventh as many.
$ python code/ch12/moe_cost.pydense 405B-like (C=16384, L=126, H=53248) total 405.8 B active 405.8 B forward ops/token ≈ 2·active = 812 GFLOP Mixtral-like (C=4096, L=32, 8 experts × H=14336, top-2) total 46.7 B active 12.9 B forward ops/token ≈ 2·active = 26 GFLOP DeepSeek-V3-like (C=7168, L=61, 256 experts × H=2048, top-8 + 1 shared) total 698.9 B active 32.7 B forward ops/token ≈ 2·active = 65 GFLOP
The Mixtral line reproduces its published sizes exactly from the shapes. The DeepSeek line lands at 699 B total against a published 671 B, because its first three layers are dense and its attention is not the plain kind (§12.4); the active count lands at 33 B against 37 B for the same reasons. The shape formula from Chapter 4, extended by one factor, gets you within a few percent of a frontier MoE.
Chapter 8's scaling laws say loss falls with parameters at fixed data. An MoE gets the parameters of a large model at the per-token compute of a small one. Empirically, an MoE with N_active active and N_total total parameters trains to a lower loss than a dense model of N_active for the same compute, and approaches (but does not reach) a dense model of N_total public, from the Switch Transformer and later reports. The intuition is the hospital's: a token about Python does not need the detectors for French poetry, so making it pay for them buys nothing. Sparsity lets capacity specialise.
What it costs. All E experts must be in memory, so an MoE is as expensive to hold as a dense model of its total size: DeepSeek-V3 needs about 1.3 TB of weights at 2 bytes, far more than one GPU (Chapter 11's expert parallelism exists for this). Routing is a hard, discrete decision, so the router's gradient is awkward (only the chosen experts get gradient; the weights gᵢ carry the signal back to the router). And the traffic must be balanced.
Left alone, routing collapses: an expert that happens to be slightly better early gets more tokens, learns more, gets better, gets more tokens. Two experts end up doing all the work while the others sit at their random initialisation. Since experts are spread across GPUs (Chapter 11), that also means two GPUs are saturated and the rest idle. Every MoE adds a balancing mechanism. The classic is an auxiliary loss that penalises uneven expert usage across a batch public (Switch Transformer). DeepSeek-V3 instead adds a per-expert bias to the routing scores, nudged up for under-used experts and down for over-used ones, outside the gradient, which avoids the auxiliary loss's interference with the main objective public. The widget's "balancing bias" toggle is that mechanism.
A second design choice: DeepSeek-V2 and V3 use many small experts (256 of width 2048) plus a shared expert that every token uses public. Fine-grained experts give the router more combinations to choose from at the same active compute; the shared expert holds what every token needs, so the routed ones can specialise harder.
Chapter 7 derived the KV-cache cost: 2 × L × (KV heads) × d values per token per conversation, and found that at long context it, not the weights, fills a serving GPU. Three variants shrink it by changing how many distinct keys and values a layer keeps.
W_K, W_V. H keys and values per token. The original design.H. Quality drops noticeably public.g groups; each group shares a key and value. Cache shrinks by H/g. Llama 3 uses g = 8 with 32 to 128 query heads public. Quality is within noise of MHA public, because keys and values turn out to be far more redundant across heads than queries are.c (DeepSeek-V3: 512), and reconstruct every head's key and value from it with a learned up-projection at attention time public. A small separate "decoupled" key of 64 dimensions carries the RoPE rotation, because rotation does not commute with the compression. The cache holds c + 64 values per token per layer, regardless of the head count.Why does MLA work when MQA does not? MQA forces every head to use the same key; MLA lets every head have its own key, but insists that all of them be linear functions of one 512-dimensional vector. That is a rank constraint, not an identity constraint, and 512 dimensions is enough room for 128 heads to disagree. DeepSeek-V2 reports a 93% KV-cache reduction against its own dense baseline with no loss in quality public. There is a computational trick too: because the up-projection is linear, it can be folded into W_Q and W_O so that attention is computed directly against the latents, and the per-head keys are never materialised at all public.
The variants above shrink the cache; they do not touch the T² arithmetic. To do that, some layers stop attending to everything.
Sliding-window attention. Each token attends only to the w tokens before it. Per layer, scores are T × w instead of T × T, and the cache for that layer can drop anything older than w. Mistral 7B used a 4,096-token window in every layer public. Information still travels further than w: a token at layer 2 reads tokens that at layer 1 read tokens w further back, so the reach after ℓ layers is ℓ·w. The widget makes the reach visible.
Local-global interleaving. Indirect reach is weaker than direct reach, and some tasks need a token at position 100,000 to read position 5 directly. Gemma 2 alternates one sliding-window layer (4,096) with one full-attention layer public; Gemma 3 uses five local layers (1,024) per global layer public. The global layers carry the long-range lookups and pay full T²; the local layers do the bulk of the work cheaply. The KV cache then splits too: local layers keep w tokens, global layers keep everything.
Attention sinks. Chapter 3 noted that heads with nothing to do dump their weight on the first token. Drop the first token from a sliding window and those heads have nowhere to dump; the model's outputs degrade sharply public (StreamingLLM). Keeping the first few tokens permanently in every window fixes it and enables effectively unbounded streaming. Some newer designs add an explicit learned sink so the softmax has an "attend to nothing" option inferred, from the pattern across recent open releases.
Sparse and learned patterns. Beyond fixed windows, a layer can attend to a learned or content-selected subset of past tokens (block-sparse, dilated, or retrieval-style patterns). DeepSeek's later work on sparse attention selects blocks by a cheap relevance score before running full attention on the selected blocks public. The theme is the same: spend T² only where a cheap test says it matters.
Chapter 4 built RoPE: each pair i of the d/2 coordinate pairs in q and k rotates by m·θᵢ at position m, with θᵢ = base^(−2i/d). The pairs form a spectrum: pair 0 turns a full circle every 2π positions, the last pair every 2π·base positions. With the classic base of 10,000 and d = 128, the slowest pair's wavelength is about 63,000 positions.
θᵢ = base^(−2i/d) rotation per position for pair i λᵢ = 2π / θᵢ positions per full turn (the wavelength) base = 10,000, d = 128: λ₀ = 6.3, λ₃₂ ≈ 630, λ₆₃ ≈ 62,000
A pair whose wavelength is longer than the trained context never completed a full turn during training. At positions beyond the trained length, it presents angles the model has never seen paired with any key, and the scores those pairs produce are garbage. Measured perplexity rises sharply just past the trained length public.
Three fixes, all in use, all public:
trained / target so the target length maps onto the trained range of angles. Every pair's wavelength stretches by the same factor, including the fast ones that encode "the previous token": after 16× interpolation, positions 1 and 2 are only 1/16 of a turn apart on the fastest pair, and nearby-token distinctions blur. Needs a short fine-tune; works to about 8–16× public.base so that only the slow pairs stretch and the fast pairs are nearly untouched. Llama 3 uses a base of 500,000 rather than 10,000 public, then continued pretraining in stages from 8k up to 128k tokens public.Two things stretching cannot fix. First, attention's arithmetic and cache still grow with T; a 1M-token context is a Chapter 10 and Chapter 19 problem as much as a positional one. Second, being able to address a position is not the same as being able to use it: "needle in a haystack" tests, where a fact is buried at a random depth in a long document, show models that address the full context but retrieve unreliably from the middle of it public. Long-context training data, not just RoPE surgery, is what closes that gap, and it is part of Chapter 13's run.
Given a parameter budget, how deep and how wide? The published designs cluster: depth grows slowly with width, aspect ratios (width over depth) in the low hundreds, and MoE designs go somewhat shallower and rely on expert count for capacity. The convergent choices of the public frontier as of the 2024–2025 releases:
| Model | Total / active | L | C | MLP | Attention | Context | Evidence |
|---|---|---|---|---|---|---|---|
| Llama 3.1 405B | 405 B / 405 B | 126 | 16,384 | dense SwiGLU | GQA 128/8, RoPE base 500k | 128k | public |
| Mixtral 8×7B | 46.7 B / 12.9 B | 32 | 4,096 | 8 experts, top-2 | GQA 32/8, sliding window 4k (from Mistral) | 32k | public |
| DeepSeek-V3 | 671 B / 37 B | 61 | 7,168 | 256 + 1 shared experts, top-8; first 3 layers dense | MLA, latent 512, RoPE key 64 | 128k (YaRN) | public |
| Gemma 2 27B | 27 B / 27 B | 46 | 4,608 | dense GeGLU | GQA, alternating local 4k / global | 8k | public |
| Qwen3 235B-A22B | 235 B / 22 B | 94 | 4,096 | 128 experts, top-8 | GQA 64/4 | 128k | inferred (from the release card; verify) |
| Closed frontier models | Widely believed to be MoE with compressed or grouped KV and long-context extension; no shapes disclosed | unknown | |||||
Beacon has so far been sized like the dense 405 B. Given the public trend, the Lab's next design decision is whether Beacon 2 should be an MoE. Here is that decision as a calculation.
Same training compute, MoE versus dense. Chapter 8's rule: training cost ≈ 6 × N_active × D. Hold the budget at Beacon's 3.8 × 10²⁵ operations.
dense Beacon N = 405 B active D = 15.6 T tokens
MoE Beacon-2 N_active = 40 B D = 158 T tokens at the same compute (10× the data, if it exists — Ch 9)
or N_active = 40 B, D = 15.6 T, and spend the other 90% of the budget elsewhere
serving, per generated token
dense: read 810 GB of weights per decode step (Ch 7)
MoE: read attention + 8 of 256 experts ≈ 2 × 37 B ≈ 74 GB per step (11× less memory traffic)
but hold 1.34 TB of weights resident (needs ≥ 8 top GPUs even idle)
KV cache at 128k, one conversation
GQA (Llama 3.1 405B shapes): 126 × 8 × 128 × 2 × 2 B ≈ 516 KB/token → 66 GB
MLA (DeepSeek-V3 shapes): 61 × (512 + 64) × 2 B ≈ 70 KB/token → 9 GB
The MoE wins on every per-token cost and loses on one thing: the floor. It must sit on enough GPUs to hold all its experts even when serving a single user, and its training run needs expert-parallel communication (Chapter 11) that a dense model does not. Every public frontier-scale release since 2024 has taken that trade public (DeepSeek, Qwen, Mixtral lineages), and the dense 405 B is the last large dense model in the public record.
Remove load balancing from an MoE. Within a few thousand steps two or three experts receive most tokens; the rest stay near initialisation and contribute nothing. Effective capacity falls toward a dense model of k experts; the GPUs holding the busy experts become the bottleneck for the whole cluster. Every published MoE reports the mechanism it uses to prevent this public.
Use MQA instead of GQA. The cache shrinks by another 8×, and quality measurably drops: keys and values are redundant across heads but not identical, and forcing one of each removes the ability of different heads to look for different things at the same position. Published ablations show GQA at 8 groups within noise of MHA and MQA clearly below it public.
Make every layer sliding-window. Per-layer cost and cache become linear in T, and the model can still reach far through indirect hops. But exact long-range lookup ("what was the variable name declared 50,000 tokens ago") requires a token to attend directly, and a chain of hops through intermediate tokens loses precision. Needle-retrieval accuracy at long range collapses; interleaved global layers restore it at a fraction of the full cost.
Drop the attention sink from a streaming window. Heads with no useful target lose their dumping ground; the softmax must put its weight somewhere, so it lands on arbitrary recent tokens and corrupts the blend. Perplexity rises sharply as soon as the first token scrolls out of the window public.
Extrapolate RoPE with no adjustment. The slow pairs present never-seen angles beyond the trained length; scores from those pairs are noise; perplexity climbs steeply within a few hundred positions of the boundary public. Every long-context release adjusts the base, interpolates, or both, and then trains on long sequences.
Use position interpolation at 64×. The fastest pairs, which distinguish adjacent tokens, are compressed until positions 1 and 2 are nearly the same angle. The model loses local word order before it gains long range. The per-pair methods (base scaling, YaRN) exist because uniform scaling hurts exactly the pairs that matter most at short range.
Say it back. Three costs shape the frontier architecture: parameters to hold, operations per token, and attention's T². Mixture of experts breaks the tie between parameters held and parameters run: replace each MLP by many expert MLPs and a router that sends each token to a few of them, so a model can be 671 billion parameters in storage and 37 billion in compute, with a balancing mechanism to stop the router collapsing onto favourites and a shared expert for what every token needs. Grouped-query attention shrinks the KV cache by sharing keys and values across groups of heads at no measurable loss; multi-head latent attention shrinks it further by storing one compressed latent per token and reconstructing every head's key and value from it. Sliding windows make most layers linear in context, with reach growing through depth, and a few interleaved global layers plus a permanent attention sink keep exact long-range lookup working. RoPE's pairs have wavelengths; those longer than the trained context never completed a turn, so extending context means raising the base or interpolating per pair, then training on long sequences. The public frontier has converged on MoE with compressed or grouped attention and staged long-context training; Beacon 2 would be built that way, trading a higher memory floor for an order of magnitude less compute and cache per token.
C = 4,096, top-2 routing, no shared expert. Compute the total and active MLP parameters for that layer. Then a router produces scores [2.1, 0.3, 1.9, −0.5, 0.0, 0.8, −1.2, 0.4] for one token: softmax them, pick the top 2, renormalise the two kept weights, and write the token's MLP output as a weighted sum. What fraction of the layer's MLP parameters did it use?moe_cost.py with a function that, for a fixed training budget of 3.8 × 10²⁵ operations and the 6ND rule, sweeps the active-parameter count from 20 B to 400 B and prints the training tokens each affords. Then add the Chinchilla-optimal token count for each active size (from Chapter 8) and mark which designs are data-limited given a 30 T-token corpus.