How a token becomes a point in space, and how a matrix moves it.
Chapter 1 left a box on the map labelled f(context; θ): integers in, a distribution out. This chapter opens the box one layer. Inside, integers become vectors, and vectors get pushed around by matrices. By the end you can read the two operations that make up almost all of a transformer's arithmetic: look up a vector and multiply by a matrix. Attention (Chapter 3) and the transformer block (Chapter 4) are arrangements of those two moves plus one bend.
The question this chapter answers: what does it mean for a model to represent a token as numbers, and what can a learned matrix do to that representation?
Describe a colour to someone who cannot see your screen. You could say "a warm, slightly dusty orange". Or you could say [230, 140, 60]: red, green, blue. The three numbers are the colour, as far as the screen is concerned.
Now notice what the numbers let you do that the words do not. Two colours are similar if their numbers are close. You can make a colour brighter by scaling all three up. You can mix two colours by adding their numbers and halving. You can ask "how orange is this?" by measuring how much of the colour points along the orange direction. None of that works on the phrase "dusty orange". All of it works on three numbers.
A language model does exactly this to tokens. Token 791, "The", is not a word inside the model. It is a list of numbers, a few thousand of them rather than three, and every question the model can ask about it is a question about geometry: how close, how far, in which direction. The numbers are not given. They are learned, and what is learned is a layout of the vocabulary in space where distance and direction mean something.
| In the picture | In the machine | The word we will use |
|---|---|---|
| Three numbers for a colour | A list of C numbers for a token, typically a few thousand | vector; C is the model width |
| The colour space | The C-dimensional space every token vector lives in | embedding space |
| Looking up a named colour in a swatch book | Fetching the vector for token id i from a table with one row per vocabulary entry | embedding lookup, table W_E |
| "How orange is this?" | Multiply matching coordinates and add up | dot product |
| Brighten, mix, rotate a colour wheel | A fixed rule that sends every vector to a new vector, keeping straight lines straight | linear map, stored as a matrix W |
| A rule that bends the space | A simple elementwise function applied after a matrix | nonlinearity (ReLU, GELU, …) |
| Expand, bend, compress | Two matrices with a nonlinearity between them | MLP (multi-layer perceptron) |
A vector is a list of numbers of fixed length. That is the whole definition. [2, 1] is a vector of length 2. A token's representation in a frontier model is a vector of length C, where C is a few thousand: Llama 3 8B uses C = 4096, the 405B model uses C = 16384 public.
Two pictures of the same list are both useful, and the book switches between them freely:
[2, 1] is the point two steps right, one step up. Closeness between points is similarity.Two operations, and only two, define what vectors can do on their own:
Add coordinate by coordinate: [2, 1] + [1, 2.5] = [3, 3.5]. Geometrically: place the second arrow at the tip of the first; the sum points to where you end up.
Scale every coordinate by a number: 3 · [2, 1] = [6, 3]. Same direction, three times the length. A negative scale flips the direction.
Length (norm): |[2, 1]| = √(2² + 1²) = √5 ≈ 2.24. Pythagoras, in any number of dimensions.
Everything a transformer does to its vectors is built from these plus the dot product below. Nothing else is needed; there is no "multiply two vectors" operation in the ordinary sense.
Adding is not a formality. In Chapter 4 you will see that every layer of a transformer adds a vector to the token's running representation. "This token is a noun" plus "this token is plural" plus "this token refers back to 'detective'" is, quite literally, a sum of arrows.
Given two vectors of the same length, multiply matching coordinates and add the results:
[2, 1] · [1, 2.5] = 2·1 + 1·2.5 = 4.5
That is the dot product. It is the single most-executed operation in a language model: attention is made of dot products, and every matrix multiplication is a grid of them. So it is worth knowing what the number means, and the answer is geometric:
a · b = |a| · |b| · cos(angle between a and b)
Read it in three cases. Same direction: the cosine is 1, the dot product is as large as the two lengths allow. Right angle: cosine 0, dot product 0, regardless of lengths. Opposite: cosine −1, dot product as negative as possible. The dot product is alignment, weighted by size.
Three-dimensional vectors, so the picture is not the whole story and the arithmetic has to carry it.
a = [ 1.0, 2.0, 0.0 ] b = [ 2.0, 1.0, 0.0 ] a·b = 1·2 + 2·1 + 0·0 = 4.0 same-ish direction c = [-2.0, 1.0, 0.0 ] a·c = 1·(−2) + 2·1 + 0 = 0.0 right angle to a d = [ 0.0, 0.0, 3.0 ] a·d = 0 + 0 + 0·3 = 0.0 right angle, different reason
Note a·c = 0 and a·d = 0 for different reasons. c lies in the same plane as a but is turned 90°. d points along an axis a does not use at all. In a 4096-dimensional space there are thousands of independent directions, so "unrelated" is the default relationship between two random vectors: their dot product is near zero. That is why dot products are informative there: a large one does not happen by accident.
import numpy as np
rng = np.random.default_rng(0)
a, b = rng.standard_normal(4096), rng.standard_normal(4096)
print(round(float(a @ b), 2), round(float(a @ a), 2)) # random pair vs self-18.14 4077.07
Two random 4096-vectors: dot product −18, tiny compared with a vector's dot product with itself, about 4000. In high dimensions, alignment is rare and therefore meaningful.
One more name. Divide the dot product by both lengths and you get just the cosine, a number in [−1, 1] that ignores size and keeps only direction. That is cosine similarity, and it is what retrieval systems in Chapter 21 use to ask "is this document about the same thing as this query?"
Now the first operation inside the box. The model holds a table with one row per vocabulary entry and C columns. Turning token id 791 into a vector means fetching row 791. That is all. No arithmetic, a lookup.
The table is called W_E and it has shape [V, C]. For Llama 3 8B that is 128,256 × 4,096 ≈ 525 million numbers, about 6.5% of the model, spent purely on "what does each token mean before any context is read" public (shape and size from the released weights).
Where do the values come from? Training. Chapter 5 shows the mechanism, but the effect is easy to state. The loss pushes tokens that occur in similar contexts toward similar rows, because if " cat" and " dog" tend to be followed by the same things, the cheapest way to predict well is to give them nearly the same vector and let the rest of the model treat them alike. Nobody tells the model that cats and dogs are both animals. The geometry falls out of the next-token bills.
Directions end up meaning things too. The famous example, from early word-vector models: the vector for king minus man plus woman lands near queen public. There is a "male → female" direction in the space and it is roughly the same arrow wherever you start. Frontier models are far less tidy than that, but the principle survives: an embedding space is a place where arithmetic on arrows corresponds to something about meaning. That is what makes adding vectors, in Chapter 4, a sensible way to accumulate what a token has learned about its context.
C has to become V scores, and that uses a matrix of shape [C, V] called the unembedding W_U. Some models make W_U the transpose of W_E to save parameters; Llama 3 does not public.Here is the second operation inside the box, and it is where the "learned" in "learned representation" does its work. A matrix is a grid of numbers. But the useful way to see it is as a function that takes a vector and returns a vector. Feed it x, get back xW. The rule is fixed by the numbers in the grid; change the numbers, change the function.
Start small. A 2 × 2 matrix acting on 2-vectors:
┌ a b ┐
x = [x₁, x₂] W = │ │ xW = [ x₁·a + x₂·c , x₁·b + x₂·d ]
└ c d ┘
└──── first ────┘ └── second ──┘
output output
Each output coordinate is a dot product between x and one column of W. So a matrix is a bundle of dot products: "how much does x align with column 1? with column 2?" The answers become the new coordinates. That is the entire mechanism, and it scales to any size.
Play with the presets and notice what never changes: grid lines stay straight, parallel lines stay parallel, evenly spaced points stay evenly spaced, and the origin does not move. That set of promises is what linear means. A matrix can stretch, squash, rotate, reflect, shear, and flatten. It cannot bend, and it cannot move the origin.
The general rule for x of length n and a matrix W of shape [n, m]:
[n] × [n, m] → [m]
x W xW (output j) = Σᵢ xᵢ · Wᵢⱼ = x · (column j of W)
The inner dimensions must match: the length of x equals the number of rows of W. The output length is the number of columns. Matrices can therefore change dimension: a [4096, 128] matrix turns a 4096-vector into a 128-vector, and a [4096, 14336] matrix turns it into a 14336-vector.
Many vectors at once. Stack T input vectors as the rows of a matrix X of shape [T, n]. Then XW has shape [T, m], and row t of the result is exactly xₜW. Same function, applied to every row independently. This is how a transformer processes a whole context in one operation: [T, C] × [C, D] → [T, D], one matrix multiplication, all tokens transformed by the same W.
Cost. Every output entry is a dot product of length n, and there are T·m entries, so a matmul costs about 2·T·n·m arithmetic operations (a multiply and an add per term). Hold on to this; Chapter 8 builds the cost of an entire training run out of it.
What is the matrix for, in a language model? Every matrix in θ is a learned change of viewpoint. The embedding gives a token a vector that means "this is ' cat'". A matrix can re-express that vector so that some coordinates now mean "how animal-like", "how likely to be a subject", "how much this needs a verb next". Which coordinates mean what is not designed; it emerges from training, because the loss rewards re-expressions that make the next token easier to predict. When Chapter 3 introduces three matrices called W_Q, W_K, W_V, each is exactly this: a learned viewpoint on the same token vector, tuned for a different purpose.
You might hope to build a whole model from matrices: embed, multiply, multiply, multiply, unembed. Bigger stack, smarter model. It does not work, and the reason is a one-line theorem: the composition of two linear maps is a linear map. Apply W₁ then W₂, and the result is the same as applying a single matrix W₁W₂. A stack of a hundred matrices is one matrix wearing a trench coat.
So a stack of matrices can only stretch, rotate, shear, and flatten the embedding space, all at once, as one rigid rule. It cannot say "if the token is like this, move it over here; otherwise leave it alone". It cannot implement a rule with an if in it. And language is nothing but conditionals: "bank" means one thing after "river" and another after "savings". A purely linear model cannot represent that, no matter how wide or deep.
The fix is almost embarrassingly small. After a matrix, apply a simple fixed function to each coordinate separately, one that is not linear. The classic is ReLU: replace every negative number with zero, leave positives alone. Modern models use smoother relatives (GELU, SiLU/Swish; Llama 3 uses SiLU inside a gated variant called SwiGLU public), but the geometry is the same: the function has a kink, and a kink is a bend the matrix could not make.
Applied coordinate by coordinate, ReLU turns each coordinate into a gate: on if positive, off if not. Put a matrix before it and each gate is a learned test, "does x point far enough along my column?" Put a matrix after it and each open gate contributes a learned vector to the output. That three-step pattern is the MLP.
The multi-layer perceptron is the component of a transformer that does most of the "thinking about one token in isolation". It is two matrices and a nonlinearity:
x [C] │ ▼ · W_in [C, 4C] expand: C numbers become 4C numbers h [4C] (Llama 3 8B: 4096 → 14336, about 3.5×) │ ▼ ReLU / GELU / SiLU bend: each of the 4C coordinates is gated a [4C] │ ▼ · W_out [4C, C] compress: back to C numbers y [C]
Why widen and then narrow? Because the width in the middle is the number of tests the MLP can run. Each column of W_in is one detector: a direction in embedding space that the unit compares the input against. Widening to 4C means four times as many detectors as the token has coordinates. Each detector that fires then adds its own row of W_out to the output. Read the MLP as: run thousands of "is it like this?" tests, and for every test that passes, add the associated vector.
A 2 → 3 → 2 MLP. Input x = [1.0, 2.0]. Three detectors in W_in (as columns), a bias per detector, and a write-back matrix W_out.
W_in (2×3) b (3) W_out (3×2)
┌ 1.0 0.0 0.7 ┐ [−0.5, ┌ 1.0 0.2 ┐
└ 0.0 1.0 −0.7 ┘ −0.5, │ 0.2 1.0 │
0.2 ] └ 0.5 −0.5 ┘
step 1 h = x·W_in + b
h₁ = 1.0·1.0 + 2.0·0.0 − 0.5 = 0.5
h₂ = 1.0·0.0 + 2.0·1.0 − 0.5 = 1.5
h₃ = 1.0·0.7 + 2.0·(−0.7) + 0.2 = −0.5 ← detector 3 says "not my pattern"
step 2 a = ReLU(h) = [0.5, 1.5, 0.0] ← detector 3 gated off
step 3 y = a·W_out
y₁ = 0.5·1.0 + 1.5·0.2 + 0.0·0.5 = 0.8
y₂ = 0.5·0.2 + 1.5·1.0 + 0.0·(−0.5) = 1.6
Now the same input with the second coordinate flipped: x = [1.0, −2.0]. Detector 2 goes silent (h₂ = −2.5), detector 3 turns on (h₃ = 0.7 + 1.4 + 0.2 = 2.3), and the output is y = [0.5·1.0 + 2.3·0.5, 0.5·0.2 + 2.3·(−0.5)] = [1.65, −1.05]. Different gates, different rows of W_out added, a different answer. That is a conditional, built from arithmetic.
import numpy as np
W_in = np.array([[1.0, 0.0, 0.7], [0.0, 1.0, -0.7]])
b = np.array([-0.5, -0.5, 0.2])
W_out = np.array([[1.0, 0.2], [0.2, 1.0], [0.5, -0.5]])
def mlp(x):
h = x @ W_in + b
a = np.maximum(h, 0) # ReLU
return a @ W_out, a
for x in ([1.0, 2.0], [1.0, -2.0]):
y, a = mlp(np.array(x)); print(x, "gates", a.round(2), "→", y.round(2))[1.0, 2.0] gates [0.5 1.5 0. ] → [0.8 1.6] [1.0, -2.0] gates [0.5 0. 2.3] → [ 1.65 -1.05]
Both lines match the hand calculation. Detector 1 stays on for both inputs (its column ignores the second coordinate), detectors 2 and 3 swap roles when the sign flips.
Now see the same idea as geometry. The widget below pushes a grid of input points through a 2 → 3 → 2 MLP. Watch the middle panel as the bend slider goes from 0 (no ReLU, pure matrices) to 1 (full ReLU).
What a frontier MLP actually learns in its detectors is a research question, but some findings are established. Individual units and small groups of them respond to recognisable features: a particular language, code syntax, a named entity, a sentiment public, from interpretability work on open models. And the MLP layers are where much factual recall appears to live: editing specific MLP weights can change what a model "believes" about a fact while leaving the rest intact public. Chapter 26 returns to both. For now the picture to keep is: the MLP is a bank of learned tests, each writing a learned vector when it passes.
| Quantity | Llama 3 8B | Llama 3.1 405B | Evidence |
|---|---|---|---|
Width C | 4,096 | 16,384 | public |
| MLP hidden width | 14,336 | 53,248 | public (≈3.5× and ≈3.25×, not exactly 4×) |
Vocabulary V | 128,256 | 128,256 | public |
Embedding table W_E | 525 M numbers | 2.1 B numbers | = V × C |
One MLP (W_in + W_out, ignoring the gate) | 117 M numbers | 1.74 B numbers | = 2 × C × hidden |
| Closed frontier models | Widths and hidden sizes not disclosed | unknown | |
What does one MLP cost per token? Two matmuls: [1, C] × [C, H] and [1, H] × [H, C], each about 2·C·H operations.
Llama 3 8B, one token, one MLP layer expand 2 × 4,096 × 14,336 ≈ 117 M ops compress 2 × 14,336 × 4,096 ≈ 117 M ops total ≈ 235 M ops per token, per layer × 32 layers ≈ 7.5 G ops per token, MLPs only
Seven and a half billion arithmetic operations to push one token through the MLPs of a "small" model. A modern GPU does on the order of 10¹⁵ per second, so this is fast, but the number is worth feeling: it is roughly two operations per parameter in the MLPs, and that "two ops per parameter per token" rule is exact enough to plan a training run with (Chapter 8).
Remove the nonlinearity. The MLP becomes x · (W_in W_out), a single [C, C] matrix. Every layer collapses into its neighbours; the entire model, attention aside, becomes one matrix from embedding to unembedding. It can learn which tokens are similar. It cannot learn a single conditional. Perplexity on real text stays near that of a bigram model.
Keep the nonlinearity but set the hidden width to C instead of 4C. Fewer detectors: about four times fewer tests per layer. The model still works, but the same parameter budget spent as a wider MLP is empirically better, which is why every published architecture widens. The exact ratio (Llama 3's ≈3.5) is a tuned trade-off, not a law public.
Freeze the embedding table at random values and train everything else. The model can still learn, because the matrices after the embedding can re-express random vectors; but it has to spend capacity undoing the randomness, and similar tokens start with no similarity to exploit. Performance drops, more for small models than large ones inferred from published ablations on random-embedding baselines. The lookup table is cheap to learn and worth learning.
Shrink the width C from 4096 to 64. Every dot product now compares 64 numbers, and 128k tokens must be laid out in a 64-dimensional space. There is not enough room for distinct directions: unrelated tokens are forced to have non-trivial dot products, and the "alignment is rare, therefore meaningful" property of §2.2 fails. Width is not decoration; it is how many independent things a token can be about at once.
Say it back. A token enters the model as an integer and immediately becomes a vector by indexing a table: row i of W_E, C numbers, learned. Vectors are points and arrows; they add and scale, and the dot product tells you how aligned two of them are, weighted by their lengths. In thousands of dimensions, alignment is rare, so a large dot product carries information. A matrix is a function on vectors: each output coordinate is the input dotted with a column, and the whole thing stretches, rotates, shears, or flattens the space but never bends it. Applied to a stack of T token vectors at once it transforms each row independently, at a cost of about two operations per matrix entry per token. Stacking matrices buys nothing, because a product of matrices is a matrix. A nonlinearity, applied to each coordinate after a matrix, adds the missing bend: a gate that opens only when the input aligns with a learned direction. The MLP is expand, bend, compress: 4C detectors, each writing back a learned vector when it fires. Beacon's tokens are vectors with sixteen thousand coordinates, and its MLPs run tens of thousands of tests on each of them, per layer.
W = [[0, 1], [−1, 0]], compute xW for x = [1, 0], [0, 1], and [2, 3]. Draw the arrows before and after. What single geometric word describes this matrix? Then compute W·W and say what it does.C = 16384, MLP hidden width 53,248, and 126 layers. How many arithmetic operations do the MLPs alone spend per token? Express it as a multiple of the MLP parameter count. Then estimate how many tokens per second one GPU sustaining 5 × 10¹⁴ useful operations per second could push through the MLPs, ignoring everything else.mlp_by_hand.py: generate 2,000 random 2-vectors, push them through the MLP, and plot input versus output (any plotting library). Then set the ReLU to the identity and plot again. Describe the difference in one sentence that uses the word "fold".