↑ Contents Appendices The book refreshes mathematics only where a mechanism needs it. This index collects those refreshers so you can find one without re-reading a chapter. Each entry states the idea in one or two lines, what it is for in the model, and where the full box lives.
Linear algebra
Idea In one line What it is for Where
Vector A fixed-length list of numbers; a point and an arrow. Add coordinate-wise; scale every coordinate; length is √(sum of squares). A token is a vector of C numbers; layers add arrows to it. Ch 2 §2.1 Dot product a·b = Σ aᵢbᵢ = |a||b| cos θ. Alignment weighted by size; near zero for random high-dimensional vectors. Attention scores; every entry of every matmul. Ch 2 §2.2 Cosine similarity The dot product divided by both lengths; direction only, in [−1, 1]. Retrieval. Ch 2 §2.2 , Ch 21 Matrix as a function xW: each output coordinate is x dotted with a column of W. Keeps lines straight and the origin fixed; can stretch, rotate, shear, flatten; cannot bend. Every learned transformation: W_E, W_Q, W_K, W_V, W_O, W_in, W_out, W_U. Ch 2 §2.4 Shapes and the row-times-column rule [T, n] × [n, m] → [T, m]; inner dimensions match; entry (t, j) = row t · column j; cost ≈ 2·T·n·m operations. Reading any layer's cost and size from its shapes. Ch 2 §2.4 Composition of linear maps W₁ then W₂ equals the single matrix W₁W₂; stacking matrices buys nothing. Why nonlinearities exist. Ch 2 §2.5 Outer product xᵀy is a matrix whose (i, j) entry is xᵢyⱼ. The gradient of a matmul with respect to its weight matrix. Ch 5 §5.4 2-D rotation (a, b) → (a cos φ − b sin φ, a sin φ + b cos φ); length-preserving; rotating two vectors by the same angle keeps their dot product. RoPE positions. Ch 4 §4.4 Root-mean-square rms(x) = √(mean of xᵢ²); dividing by it resets a vector's size and keeps its direction. RMSNorm. Ch 4 §4.3 Low-rank factorisation A [n, m] update written as BA with B [n, r], A [r, m], r ≪ n, m: far fewer numbers, restricted directions. LoRA adapters. Ch 14
Probability and information
Idea In one line What it is for Where
Probability distribution over a finite set V non-negative numbers summing to 1. The model's only output. Ch 1 §1.2 exp and ln e^z is always positive, order-preserving, and turns sums into products; ln is its inverse and turns products into sums. Softmax; sequence loss as a sum. Ch 1 §1.3 Softmax pᵢ = e^{zᵢ/T} / Σⱼ e^{zⱼ/T}; positive, sums to 1, order-preserving, amplifies gaps; T is temperature. Logits to probabilities; attention scores to weights. Ch 1 §1.3 , Ch 3 §3.4 Surprise / cross-entropy −ln p(true outcome): 0 when certain and right, unbounded when certain and wrong, additive over a sequence, a proper scoring rule. The training loss. Ch 1 §1.4 Perplexity e^{mean loss}: the effective number of equally likely choices. Reading loss curves. Ch 1 §1.4 Nats and bits ln gives nats; divide by ln 2 for bits. Comparing papers' units. Ch 1 §1.4 Variance of a sum A sum of d independent unit-variance terms has spread √d. Why attention divides by √d. Ch 3 §3.4 Averaging cancels noise The mean of B noisy estimates has spread shrinking like 1/√B. Minibatches; eval sample sizes. Ch 5 §5.5 , Ch 18 Bradley–Terry P(A beats B) = σ(r_A − r_B): pairwise preferences from scalar scores. Reward models; arenas. Ch 15 , Ch 18 KL divergence Σ p ln(p/q): how far distribution p has moved from q; zero only when equal. The RLHF penalty; distillation. Ch 15 , Ch 27
Calculus
Idea In one line What it is for Where
Derivative as slope dL/dw ≈ (L(w+ε) − L(w))/ε; sign says which way is downhill, size says how steep. Which way to move each parameter. Ch 5 §5.1 Basic rules d(aw)/dw = a; d(w²)/dw = 2w; d(ln w)/dw = 1/w; d(e^w)/dw = e^w; derivative of a sum is the sum of derivatives. Every local derivative in the book. Ch 5 §5.1 Gradient The vector of partial derivatives, one per parameter; points uphill; step against it. The update θ ← θ − η∇L. Ch 5 §5.1–5.2 Chain rule dL/dw = dL/dz · dz/dw; along a chain, multiply; along parallel paths, add. Backpropagation. Ch 5 §5.3 Local derivatives of the model's pieces Softmax + cross-entropy: p − onehot. Matmul: outer product and Wᵀ. ReLU: a 0/1 gate. Residual add: exactly 1 along the highway. Why each component is shaped the way it is. Ch 5 §5.4 Numeric gradient check Nudge one input by ε, measure the loss change, divide, compare with the formula. Testing any gradient implementation. Ch 5 §5.4 , Ch 7 §7.5 Adam Running averages of the gradient (m) and its square (v); step −η·m/√v per parameter. The optimiser every frontier run uses. Ch 5 §5.5
Counting and estimating
Rule In one line Where
Parameter count L·(2C² + 2C²·r + 3CH) + 2VC for a gated, grouped-query transformer; reproduces Llama 3 sizes. Ch 4 §4.6 2N rule Forward ≈ 2 operations per parameter per token, plus attention's T² terms. Ch 4 §4.7 6N rule Forward + backward ≈ 6 operations per parameter per token; 6·N·D for a run of D tokens. Ch 5 §5.4, §5.7 , Ch 8 16 bytes per parameter Weight (2) + gradient (2) + fp32 master (4) + Adam m, v (8) in mixed precision. Ch 5 §5.7 KV cache per token 2 × L × (KV heads) × d × bytes. Ch 7 §7.2 Decode is memory-bound Time per step ≈ bytes of weights ÷ memory bandwidth, not FLOPs ÷ compute. Ch 7 §7.2 , Ch 19 Embedding tables 2·V·C parameters; the vocabulary-size trade-off. Ch 6 §6.5