Shapes are written in brackets, batch first. Inner dimensions must match to multiply: [n] × [n, m] → [m], and [T, n] × [n, m] → [T, m] applies the same map to every row.
Tensor
Shape
What it is
token ids
[B, T]
integers from the tokenizer
x, the residual stream
[B, T, C]
one vector per token, the same shape through every block
Q, K, V
[B, T, d] per head
queries, keys, values
scores, attention weights
[B, T, T] per head
Q·Kᵀ/√d, masked, softmaxed per row
logits z, probabilities p
[B, T, V]
one next-token distribution per position
KV cache
[L, 2, KV heads, T, d] per sequence
stored keys and values of every past token
Learned matrices
Matrix
Shape
Role
Introduced
W_E
[V, C]
embedding: token id selects a row
Ch 2 §2.3
W_Q, W_K, W_V
[C, d] per head
query, key, value projections
Ch 3 §3.2
W_O
[C, C]
mixes concatenated heads back into the stream
Ch 4 §4.1
W_in, W_out
[C, H], [H, C]
MLP expand and compress (plus W_gate [C, H] when gated)
Ch 2 §2.6
g
[C]
RMSNorm gains
Ch 4 §4.3
W_U
[C, V]
unembedding: stream to logits
Ch 2, Ch 4
Quantities and operations
Symbol
Meaning
Introduced
θ
all the model's parameters
Ch 1
z
logits: unconstrained scores, one per vocabulary entry
Ch 1 §1.3
p = softmax(z / T)
probabilities; T here is temperature, not sequence length
Ch 1 §1.3
loss = −ln p(target)
cross-entropy: the surprise at the true next token, averaged over positions
Ch 1 §1.4
e^(mean loss)
perplexity
Ch 1 §1.4
a · b
dot product, Σ aᵢbᵢ = |a||b|cos θ
Ch 2 §2.2
∂L/∂w, ∇L
partial derivative; gradient (same shape as θ)
Ch 5 §5.1
η
learning rate
Ch 5 §5.2
m, v
Adam's running averages of the gradient and its square
Ch 5 §5.5
KL(p ‖ q)
Σ p ln(p/q): how far q is from p
Ch 15, Ch 27
β
strength of the KL penalty in preference optimisation