Interpretability: what we can actually see in a trained model, how we see it, and how much to trust it.
Part 1 built the machine and Parts 2 and 3 trained it, but every claim about what the trained numbers mean was made from the outside: the loss went down, the benchmark went up, the assistant behaves. This chapter turns the model over and looks at the numbers themselves. It leans on things you already own: the residual stream as a shared bus (Chapter 4), heads with jobs (Chapter 3), MLP detectors (Chapter 2), gradients as blame (Chapter 5). It also closes a loop opened in Chapter 17: if alignment is about what a model will do, interpretability is the only tool that asks why, from the inside.
The question this chapter answers: when researchers say a model "has a feature for X" or "uses a circuit for Y", what did they measure, what does it prove, and where does it stop?
You inherit a factory that runs perfectly and comes with no documentation. Four hundred billion dials, all set, humming. You would like to know what a particular dial does before anyone touches it.
There are three things you can do, and they are increasingly convincing. You can watch: put a meter on a wire and see when it carries current, and notice that it lights up whenever a red widget passes. You can correlate: train a small gauge that predicts "red widget on the line" from the wire's readings and see how well it does. Or you can intervene: cut the wire, or splice in the reading from a different moment, and see whether the widgets downstream come out wrong. Only the third tells you the wire matters. The first two tell you the information is there.
And there is a complication the factory metaphor hides. The wires do not each carry one signal. The factory has far more signals than wires, so the signals are superimposed, at angles, and a single wire lights up for red widgets and for Tuesday deliveries and for a particular supplier. Reading one wire tells you a smeared mixture. To read cleanly you need to find the angles first.
| In the picture | In the machine | The word we will use |
|---|---|---|
| A wire | One coordinate, or one direction, of the residual stream at a layer; or one MLP unit | neuron / direction |
| A meter on a wire | Recording activations on many inputs and looking at what they respond to | activation analysis |
| A small gauge trained to read one property | A linear classifier on the residual stream | linear probe |
| Cutting or splicing a wire | Overwriting an activation with another value and re-running downstream | activation patching, ablation |
| A set of wires and machines that together do one job | Specific heads and MLP units, connected through the stream, that implement a behaviour | circuit |
| More signals than wires | More features than dimensions; features stored at angles, interfering | superposition |
| A wire that lights up for unrelated things | A neuron that responds to several unrelated features | polysemanticity |
| Finding the angles so each signal reads cleanly | Learning an overcomplete dictionary of directions with a sparsity penalty | sparse autoencoder (SAE), feature |
| Forcing a signal onto the wires | Adding a feature's direction to the stream during generation | steering |
Everything a trained model does is arithmetic on tensors you can print. That is the whole promise of the field: unlike a brain, the model is fully observable. For a given input you can read, at every layer, the residual stream [T, C], every head's attention pattern [T, T], every MLP unit's activation [T, H], and the gradient of any output with respect to any of those. And you can change any of them and re-run.
The difficulty is not access. It is that the numbers are not organised for reading. Chapter 4 called the residual stream a bus that components communicate over; the bus has no labelled lanes. Chapter 2's MLP detectors are "learned directions", and nothing forces a direction to align with a coordinate, or to mean one thing. So the field has three layers of method, from cheapest to most conclusive:
[T, T] matrix of weights that you can plot over the tokens. This is how previous-token and induction heads were first noticed (Chapter 3) public. For MLP units, "look" means: which tokens in a large corpus make this unit fire hardest, and do they have something in common.A linear probe is logistic regression on the residual stream. Freeze the model, collect the vector at layer ℓ for many tokens, label each token with the property (inside a quoted string; the subject is plural; the current chess position is in check), and fit a weight vector w and a threshold. If held-out accuracy is high, the property is linearly available at that layer. Probes have found positions on a board game the model was only ever shown as move text, the truth-value of statements, and the model's own eventual answer, well before the final layer public.
The script trains a probe on an 8-dimensional toy stream where the property was written along one hidden direction plus noise, then steers: it subtracts along the probe direction and watches the readout flip.
$ python code/ch26/linear_probe.pyprobe accuracy on held-out tokens: 0.88 cosine between probe direction and the true direction: 0.96 token 0 label=1, readout before: 1 after subtracting 3 units along the probe direction, readout: 0
Read the second line. The probe recovered the direction the property was written along (cosine 0.96 with the truth), from labels alone, never having seen the direction. That is what probes do well. Read the last two lines with suspicion: the readout flipped because we moved along the probe's direction, and the probe's readout is by construction sensitive to it. Whether the model's downstream layers would change their behaviour is a different question, and answering it needs §26.5.
Two known failure modes make probes evidence rather than proof. A probe can succeed by reading a correlate of the property rather than the property (tokens inside quotes are also more often nouns). And a probe with enough parameters can fit anything: that is why the field insists on linear probes and on held-out accuracy well above a baseline. A high-accuracy linear probe says "the model has arranged this information so that a single direction reads it". It does not say the model reads it that way.
Chapter 3's induction head is the founding example of a circuit. It takes two heads in two layers: a previous-token head in an early layer writes "the token before me was X" into each position's stream, and an induction head in a later layer uses that as its key to find the earlier position whose predecessor matches the current token, then copies what followed. Neither head does the job alone; the composition through the residual stream does. The pattern was found by looking at attention patterns, confirmed by ablating each head (the copying behaviour disappears), and shown to appear abruptly during training with a visible bend in the loss curve public.
The larger worked example in the literature is the indirect object identification circuit in GPT-2 small: for "When Mary and John went to the store, John gave a drink to …", the model predicts "Mary", and a 26-head circuit was mapped that finds the duplicated name, suppresses it, and copies the other one to the output, with each head's role established by patching public. It took months for a two-sentence behaviour in a 124M model. Circuits are real, discoverable, and expensive to find by hand, which is why the rest of the chapter is about tools that scale.
Early hope was that individual MLP units would be the features: one unit for "Python code", one for "a name". Looking (§26.1) found some such units, and many more that fire for a name and a chess opening and a kind of cheese. The explanation is geometry, and it is the single most important idea in the chapter.
A layer of width C has C orthogonal directions. If the model wants to represent more than C features, it cannot give each its own clean axis. But most features are sparse: on any given token, only a few of the thousands of possible properties are true. Sparse features can share a space by sitting at angles: two directions at 80° interfere a little when both are active, but if they are almost never active together, the interference costs almost nothing, and the model gains a feature. The result is far more features than dimensions, each smeared across many coordinates, and each coordinate (each neuron) carrying a smear of many features. That is superposition, and polysemantic neurons are its symptom public (Anthropic's toy-model paper demonstrated it in exactly the setting below).
$ python code/ch26/superposition.pyfeatures active with prob 1.0 : features represented = 2/5 norms = [1. 0.99 0.17 0.1 0.02] loss = 0.025 features active with prob 0.3 : features represented = 4/5 norms = [1.02 1.03 1. 0. 0.98] loss = 0.015 features active with prob 0.05: features represented = 5/5 norms = [0.97 1.02 0.99 0.46 0.99] loss = 0.002
The numbers in the script tell the whole story in three lines. Dense features: the two most important survive, three are dropped. Make features rare and the same two dimensions carry all five, with lower total loss, because interference between rarely co-occurring features is cheap. A frontier model's stream is 16,384 wide and its "features" are the concepts in fifteen trillion tokens of text. By this argument there is no hope that coordinates mean things. Directions do, and finding them is the next problem.
If features are sparse directions in superposition, then a good dictionary of directions should let you rewrite any activation vector as a sparse combination of dictionary entries. That is a sparse autoencoder: an encoder that maps a C-dimensional activation to F ≫ C feature activations with a penalty for being non-zero, and a decoder that reconstructs the original from them. Train it on billions of activation vectors from one layer, and the decoder's columns are candidate features: directions that, when active, correspond to something.
Published results at scale found features that are strikingly specific and often multilingual and multimodal: a feature for the Golden Gate Bridge that fires on its name in several languages and on images of it; features for code errors, for a particular kind of deception in text, for famous people, for syntactic structures public. And the decisive demonstration was intervention: clamping the bridge feature high during generation made the model steer every conversation toward the bridge, which established that the direction is used, not merely present public.
encode f = ReLU( (x − b_dec) · W_enc + b_enc ) F features, F ≫ C (e.g. 16× to 256× wider) decode x̂ = f · W_dec + b_dec each row of W_dec is one feature direction loss |x − x̂|² + λ · Σ|fᵢ| reconstruct well, using as few features as possible
The ReLU makes activations non-negative; the L1 penalty makes most of them exactly zero on any given token. Chapter 2's "detector" reading of an MLP unit applies to each encoder row: a feature fires when the activation points enough along its encoder direction. The trade-off is set by λ: a strict penalty gives cleaner features and worse reconstruction. Reconstruction is never perfect, and what the SAE fails to reconstruct is behaviour it cannot explain.
The convincing method. Run the model on a clean input and on a corrupted one that differs in the property of interest; record every activation from both. Then re-run the corrupted input while overwriting one chosen activation (one layer's stream at one position, one head's output, one feature) with its value from the clean run, and measure how much the output moves back toward the clean answer. Where the patch restores the answer, that activation carried it. Sweep the position and layer and you get a map of where the answer lives and when it was computed.
Three variants you will see named. Ablation replaces an activation with zero or with its mean over a corpus, asking "what breaks without this?"; it is cruder than patching because it takes the model off-distribution. Attribution uses the gradient of the output with respect to an activation (Chapter 5's machinery pointed at an internal quantity rather than a weight) as a cheap linear estimate of what patching would find; it is fast, runs everywhere at once, and is wrong when effects are non-linear. Attribution graphs combine SAE-style features with attribution to draw, for one prompt, the graph of which features caused which, and published examples show multi-step "reasoning" inside a single forward pass, such as retrieving an intermediate fact on the way to an answer, and cases where the model's stated reasoning did not match its internal one public (Anthropic, 2025, on a mid-sized production model).
An honest ledger, because this field attracts overstatement in both directions.
| Claim | Status | Basis |
|---|---|---|
| Trained models contain legible heads (previous-token, induction, copy-suppression) and their composition is a real mechanism | established | patterns, ablation, training dynamics, many replications public |
| Features are directions in superposition; neurons are polysemantic; sparse dictionaries recover many interpretable features at frontier-adjacent scale | established at the scales published; completeness unknown | toy models, SAEs on Claude 3 Sonnet, Gemma, GPT-2 public |
| Steering a feature changes behaviour in the expected direction | established for specific features | clamping demonstrations public |
| Full circuits for specific behaviours can be mapped | established for small models and narrow tasks; partial for large ones | IOI, attribution graphs public |
| A frontier model's overall behaviour can be explained end to end | not achieved | SAE reconstruction error and unexplained variance remain large public |
| What closed labs use interpretability for internally, beyond what they publish | — | unknown |
Why it matters, beyond curiosity: it is the only channel to the questions Chapter 17 could not settle from outside. Whether a model that says it is being helpful is computing something else; whether a refusal is a learned rule or a learned performance; whether a feature for "the user is testing me" exists and is active. Published work has found features for deception-related concepts and shown that stated reasoning and internal computation can diverge public. None of that is a verdict on any deployed model; it is the reason the tools are being built.
| Quantity | Value | Evidence |
|---|---|---|
| Residual width at a mid layer | 16,384 (405B-class) | public (Llama 3) |
| SAE dictionary sizes published | ≈ 1M to 34M features on a mid layer of a production model | public (scaling monosemanticity) |
| Activations needed to train one | ≈ billions of tokens' worth of one layer | public |
| Fraction of a layer's variance an SAE explains | high but not complete; residual unexplained | public, exact figures vary by setup |
| Beacon's own interpretability programme | — | unknown |
What does it cost to run one SAE on one layer of Beacon? The autoencoder is two matrices, C × F and F × C.
C = 16,384, F = 4M features W_enc + W_dec = 2 × 16,384 × 4M ≈ 1.3 × 10¹¹ parameters (a third of Beacon itself) training data: 4B tokens × 16,384 × 2 bytes ≈ 130 TB of activations to stream once per-token cost of encode + decode: 2 × 2 × C × F ≈ 2.6 × 10¹¹ ops (about a third of a Beacon forward pass, for ONE layer)
A dictionary for every layer would be several Beacons' worth of parameters. This is why published SAEs cover a few layers, and why the field also works on cheaper attribution methods. Reading a frontier model is, so far, comparable in cost to building one.
Trust max-activating examples alone. The tokens that make a unit fire hardest are the tail of its distribution. A unit whose top ten activations are all chess notation may fire on a thousand other things at moderate strength. Many early "we found a chess neuron" claims dissolved under this check. Observation suggests; it does not establish.
Use a deep non-linear probe. Accuracy rises; meaning falls. A two-layer probe can compute the property from raw information the model never organised, so success no longer says the model represents it. Keep probes linear and compare against baselines (a probe on a randomly initialised model, or on shuffled labels).
Skip the intervention and steer along a probe direction. The readout flips (the by-hand example showed it) and nothing downstream need change. Probe directions and the directions the model reads from can differ; only patching or steering with a measured behavioural effect closes the gap.
Train the SAE with no sparsity penalty. With F ≫ C the autoencoder can reconstruct perfectly using dense combinations of meaningless directions. The sparsity penalty is what forces the dictionary to find the model's own sparse features rather than any basis at all.
Read a circuit found in a 124M model as the mechanism in a 400B one. Induction heads do transfer; the IOI circuit's exact heads do not. Larger models have more, more redundant, and more distributed circuits, and the published attribution graphs for production models look different from GPT-2's. Scale changes mechanism, not only accuracy.
Say it back. A trained model is fully observable: every activation, pattern, and gradient can be read and changed. The problem is organisation, not access, because the stream stores more features than it has dimensions, in superposition, so coordinates are polysemantic and meaning lives in directions. Three levels of method: observe activations and attention patterns; read out properties with linear probes, which establish that information is present as a direction; intervene by patching, ablating, or steering, which establishes that the model uses it. Circuits are specific heads and units whose composition through the residual stream implements a behaviour, found first by observation and confirmed by intervention; the induction circuit is the canonical one and the IOI circuit the canonical hard-won one. Sparse autoencoders learn an overcomplete dictionary of directions under a sparsity penalty, recovering many specific, interpretable, steerable features from a layer, at a cost comparable to a slice of the model itself, and with a reconstruction gap that bounds what they explain. Attribution methods make intervention cheap and approximate, and attribution graphs have shown multi-step computation inside one forward pass. The field has established mechanisms, features, and steering at published scales; it has not explained a frontier model end to end; and it is the only route to asking what a model is computing when it tells you what it is doing.
F = 16 × C is trained on layer 40 of a model with C = 8,192 using 2B tokens. How many parameters does it have, how many bytes of activations must be streamed at 2 bytes each, and how many operations does training cost if each token costs 6 × (parameters) as in Chapter 5? Compare with the cost of pretraining a 7B model on 2B tokens.linear_probe.py: write the property along a non-linear rule (label = 1 if the product of coordinates 1 and 2 is positive) and show that the linear probe fails while a probe on the elementwise product of pairs succeeds. Then extend superposition.py to 8 features in 3 dimensions and plot the learned norms against sparsity.