Where the machine you now understand is stuck, and where it is moving.
You started with a function that returns a distribution over the next token (Chapter 1). You now know what is inside it, how its numbers are found, how a lab makes one at the scale of Beacon, how it becomes an assistant, how it is served, and how to build something on it that you can measure and operate. This closing chapter does not add a mechanism. It uses the ones you have to look at the questions nobody has answered yet, because the fastest way to understand where the field is going is to know exactly which parts of the machine are under strain.
Each problem below points back to the chapter that set it up. Facts about the current state of particular models live in Appendix B, dated, because they will be out of date soon. The problems will not.
Chapter 8's scaling laws say loss falls predictably with more parameters and more data, and Chapter 9 counted the data. Frontier runs already train on tens of trillions of tokens, and the stock of high-quality public human text is of the same order public (Villalobos et al., 2024 estimate). Repeating data helps for a few epochs and then stops helping public (Muennighoff et al., 2023). So the question is what replaces fresh text. The candidates are all in this book: synthetic data from strong models (Chapter 9), which risks models learning their own blind spots; more modalities (Chapter 25), which add tokens but not obviously the same kind of knowledge; and reinforcement learning with verifiable rewards (Chapter 16), which generates its own training signal wherever answers can be checked. Nobody knows yet how far each goes, and in particular whether training on model-generated data degrades a model over generations, as small experiments suggest it can public (Shumailov et al., 2024), or whether careful filtering makes that a non-issue at frontier scale unknown.
Chapter 4 ended with a constraint: a token's vector passes through each block once. There is no loop inside the model, so the only way to spend more computation on a hard problem is to generate more tokens. Chapter 16 showed reasoning models doing exactly that, and doing it well. But it is an odd architecture for thought. Every intermediate step must be squeezed through a single sampled token from a vocabulary built for human text (Chapter 6), and the long reasoning traces cost decode time and KV cache linearly (Chapter 7). Open questions: whether reasoning can happen in continuous vectors instead of tokens; how much of a written chain of thought actually reflects the computation that produced the answer, which matters for anyone trying to read it (Chapter 26); and how to allocate test-time compute, spending it where it helps and not on easy questions.
Part 3 turned a text predictor into an assistant by rewarding what people prefer (Chapter 15), what verifiers accept (Chapter 16), and what a written constitution describes (Chapter 17). Each of these is a proxy, and Chapter 15 drew the curve that shows what happens when a proxy is optimised hard: it rises while the thing it stood for falls. Sycophancy, telling people what they want to hear, is the everyday form of that failure, and it is still measurable in current models public. The deep problem is that the behaviour we want is hard to write down completely, and the models are now capable enough to find the gaps in what we did write down. Progress is real: rubrics, AI feedback, and evaluations for specific failure modes all help. But there is no method yet that guarantees a model's goals match the specification, rather than approximating it on the situations that were tested.
Chapter 22 showed the arithmetic that limits agents: a per-step success rate of 95% gives about a 60% chance of getting through ten steps, and long tasks have dozens of steps. Models have improved at each step and at recovering from their own errors, and agents now complete tasks that took hours of human work in controlled evaluations public (published agent benchmarks). The open problems are the ones that separate a demo from a colleague: knowing when to stop and ask, not taking an irreversible action on a misreading, resisting instructions hidden in the documents they read (Chapter 17's prompt injection), and being evaluated fairly, since an agent's environment changes under it and a transcript is not an end state (Chapter 23 §23.6). The gate, the budget, and the approval queue of Chapters 22 and 24 are today's answer. They work by keeping a human in the loop, which is exactly what makes agents slower than their promise.
Chapter 26 showed what interpretability can do: find features in a model's activations, trace small circuits like induction heads, and steer behaviour by adding a direction to the residual stream. It also showed the scale of what is left. The features found so far explain part of a model's computation, not all of it, and methods that work on one layer or one behaviour do not yet add up to an account of why a frontier model gave a particular answer. The stakes are practical. If you could read whether a model is being deceptive, or whether its stated reasoning is its real reasoning, many of the safety problems of Chapter 17 would become engineering problems. Whether interpretability reaches that point before models become much more capable is one of the field's genuinely open races.
Chapter 18 described the benchmark cycle: a benchmark is published, models improve on it, it saturates, and a harder one replaces it. The cycle is now measured in months. Contamination makes old benchmarks unreliable, and the hardest new ones require expert humans to write and grade. Meanwhile the capabilities that matter most in deployment, like whether an agent does the job without supervision or whether a model's advice is safe, are the hardest to put in a benchmark. Chapter 23's advice applies to the whole field: your own cases, written from real failures, with graders you have checked, are worth more than any leaderboard. The field's version of that advice, evaluations that keep pace with the models and measure what matters rather than what is easy, does not yet exist at scale.
Every generated token reads every weight (Chapter 7) and every training token costs six operations per parameter (Chapter 5). Chapter 27's techniques and Chapter 19's serving tricks have cut the cost of a token of given quality dramatically, and the demand for tokens has grown faster: reasoning models spend thousands of tokens per answer, and agents spend them in loops. The frontier training runs of Chapter 13 are now bounded by power and data-centre construction as much as by chips public (lab and industry statements). The open problems here are hardware and architecture together: models that do less work per token (mixture-of-experts, Chapter 12, is one step), memory that moves weights less far (Chapter 10), and whether the scaling of Chapter 8 continues to be worth its bill.
A last question is not technical, and it shaped this book. Almost every number in Beacon's tables came from open-weight models: Llama, DeepSeek, Qwen, Gemma, OLMo, Mistral. The most capable closed models publish much less, which is why so many claims about them carry the unknown marker. The gap between the best open and best closed models has varied from many months to a few inferred from public benchmark histories. How that gap evolves decides whether the next edition of a book like this one can be written from public evidence at all.
You now have the mechanism under every one of these problems. When a paper claims progress on one of them, you can ask the questions this book taught: which part of the machine does it change? What is the bill, and what does its gradient push toward? What does it cost per token, at training and at inference? How was it measured, on how many cases, against what noise? Those questions will outlast every specific model named in Appendix B.