↑ Contents Appendices token a chunk of text from a fixed list; the unit the model reads and writes. Seen by the model as an integer id. vocabulary (V) the fixed list of all tokens a model can output; ≈128k for current frontier models. context the sequence of token ids the model can see when predicting the next one. distribution (over the next token) V non-negative numbers summing to 1; the model's only output type. logits the V unconstrained scores the model produces before softmax. softmax exponentiate each score, divide by the sum; turns logits into a valid distribution, amplifying the leader. temperature (T) divide logits by T before softmax; T<1 sharpens, T>1 flattens; a sampling-time knob, not learned. target the token that actually came next in the text. loss (cross-entropy) −ln p(target); the surprise; zero when certain and right, unbounded when certain and wrong; additive over a sequence. perplexity e^(mean loss); the effective number of equally likely choices the model was facing. autoregression generate by sampling one token, appending it, and predicting again. greedy vs sampling take the argmax, or draw with the model's probabilities as odds; a property of the sampler, not the model. parameters (θ) the adjustable numbers inside the model; the only place knowledge lives; frozen at inference. training billing the model on real text and nudging θ to lower the bill. inference running the frozen model to produce text. stop reason why generation ended: the model sampled an end token (end_turn) or the caller's limit was hit (max_tokens). Beacon / the Lab the book's imagined frontier model and its builder (lab spine). Dispatch / Postbox the book's product, an on-call assistant, and the fictional service it serves (builder spine). vector a fixed-length list of numbers; a point and an arrow; the form every token takes inside the model. width (C) the length of a token's vector; the model dimension (Llama 3 8B: 4096). embedding space the C-dimensional space token vectors live in, where distance and direction carry meaning. embedding table (W_E) a [V, C] matrix; token id i selects row i. Learned. unembedding (W_U) a [C, V] matrix at the end of the model that turns a vector into V logits. dot product sum of coordinate-wise products; equals |a||b|cos θ; alignment weighted by size. cosine similarity dot product divided by both lengths; direction only. matrix a grid of numbers read as a function on vectors; each output coordinate is a dot product with a column. linear map a function that keeps straight lines straight and fixes the origin; every matrix is one, and compositions of matrices are still one. shape the dimensions of a tensor, written [rows, columns]; inner dimensions must match to multiply. nonlinearity / activation an elementwise function with a bend (ReLU, GELU, SiLU) applied after a matrix; supplies the conditionals. ReLU max(0, z); a gate that passes positives and zeroes negatives. MLP expand (W_in, C→4C), bend (activation), compress (W_out, 4C→C); a bank of learned detectors that write learned vectors. hidden width the MLP's middle dimension; the number of detectors. self-attention every token is a query, a key, and a value; each token's output blends the values of the tokens whose keys match its query. query (q) x·W_Q; what a token is looking for. key (k) x·W_K; what a token advertises to queries. value (v) x·W_V; what a token hands over when attended to. head width (d) the length of q, k, v for one head (Llama 3: 128). score q·k for a query–key pair; the [T, T] matrix Q·Kᵀ holds all of them. scaled dot-product attention scores divided by √d before softmax so their spread does not grow with head width. attention weights softmax of each row of scaled, masked scores; positive, summing to 1 per query. causal mask set scores where the key follows the query to −∞, so no token sees its future. attention output weights · V; a blend of values inside their span. Attention routes; it does not create. previous-token head / induction head / attention sink legible jobs found in trained heads: copy position i−1; copy what followed an earlier occurrence; dump weight on the first token. block (layer) attention sub-layer then MLP sub-layer, each pre-normed and added to the stream; the repeating unit. multi-head attention H heads of width C/H in parallel, outputs concatenated and mixed by W_O; H independent weightings per layer at no extra parameter cost. W_O the [C, C] output projection that routes each head's slot into any direction of the stream. grouped-query attention (GQA) fewer KV heads than query heads; groups of queries share keys and values. residual stream the token's [C] vector as it flows through the stack, growing by addition; a shared workspace for all components. residual connection x ← x + f(x); nothing is erased; also the clean path for gradients. RMSNorm divide by root-mean-square, multiply by learned gains; resets size, keeps direction. pre-norm normalise the copy entering each sub-layer, leave the highway untouched (the modern default). RoPE rotate q and k pairs by position × base angle inside each head; scores then depend only on the distance between tokens. forward pass embed → L blocks → norm → W_U → softmax; produces a next-token distribution at every position at once. 2N rule a forward pass costs about two operations per parameter per token, plus attention's T² terms. parameter / weight one adjustable number in θ; a dial. loss landscape the loss as a function of θ for fixed data. derivative / slope how much the loss changes per unit change of a quantity, for tiny changes. gradient (∇L) the vector of partial derivatives, one per parameter; same shape as θ; points uphill. chain rule dL/dw = dL/dz · dz/dw; blame passes through a chain by multiplying local derivatives; contributions along parallel paths add. backpropagation the chain rule organised as one backward walk over the kept forward values; every gradient at about twice the forward cost. local derivative one operation's output sensitivity to its input: p − onehot (softmax + CE), outer product (matmul), a gate (ReLU), exactly 1 (residual highway). learning rate (η) the step size; too small is slow, too large diverges; warmed up, held, decayed. training step forward, loss, backward, update. minibatch the tokens whose losses are averaged for one step; millions at frontier scale. Adam / AdamW per-parameter step sizes from running averages of the gradient and its square; two extra numbers per parameter; plus weight decay. 6N rule forward + backward ≈ six operations per parameter per token. 16 bytes per parameter weight, gradient, master copy, and two Adam moments in mixed precision. subword tokenization units between characters and words: common strings are one token, rare ones several; closed vocabulary, open coverage. byte-pair encoding (BPE) repeatedly merge the most frequent adjacent pair; the ordered merge list is the tokenizer. merge list the learned sequence of pair merges, applied in order to tokenize new text. byte-level BPE base vocabulary is the 256 byte values, so any string is representable; rare characters cost one token per byte. pre-tokenization a regex split (spaces attached to the following word, punctuation, digit runs ≤ 3) that merges may not cross. special tokens reserved ids for structure (begin/end of text, turn markers) that ordinary text can never produce. tokens per word / bytes per token the compression ratio; sets context capacity and cost; worse for underrepresented languages and domains. glitch token a vocabulary entry so rare in training that its embedding row was never updated. vocabulary-size trade-off larger V: shorter sequences, larger tables (2·V·C), costlier unembedding, more undertrained rows. teacher forcing during training, every position is conditioned on the true preceding tokens, so all positions train in parallel from one pass. parallel training one forward pass yields T distributions and T bills; one backward pass updates everything. prefill the prompt processed in one parallel pass; compute-bound; produces the first token and the initial KV cache. decode one forward pass per generated token over one new position; memory-bound (weights read once per step). KV cache stored keys and values of every past token at every layer; makes decoding linear; 2·L·(KV heads)·d·bytes per token. memory-bound vs compute-bound decode is limited by reading the weights, prefill by arithmetic; batching amortises the read. top-k / top-p / min-p truncation rules applied to the model's distribution before sampling, then renormalised. stop token / max_tokens how a turn ends: the model samples end-of-turn, or the caller's budget is hit. context vs parameters conversations change the context; only a training loop changes θ. scaling law held-out loss as a power law in parameters, tokens, or compute; L = E + A/N^α + B/D^β with a floor. irreducible loss (E) the entropy of the text; the floor no scale beats (≈1.69 nats in the Chinchilla fit). compute-optimal (Chinchilla) for a fixed budget C = 6ND, the N and D that minimise loss; tokens grow with parameters, tens to over a hundred per parameter. tokens per parameter (D/N) the ratio that summarises a run's balance; ≈20 in the simple rule, rising with budget in the full fit; 1,875 for Llama 3 8B. over-training training a smaller-than-optimal model far past its optimal token count to cut serving cost (2N per token, forever). MFU (model FLOPs utilisation) sustained useful FLOP/s divided by peak; 35–45% in large runs. emergence a benchmark score that jumps with scale while loss moves smoothly; often a metric artefact (p^k), sometimes not. refit a lab's own scaling-law fit on its own data before a run; the first page of a run plan. raw crawl petabytes of HTML from Common Crawl or a private crawler; the intake, never used directly. extraction HTML to main-content text; quality here matters more than expected, especially for math and code structure. language identification a fastText classifier routing documents by language with a confidence threshold. heuristic quality filters cheap rules (length, symbol ratio, repetition, stopwords, boilerplate) that remove obvious garbage; Gopher and C4 are the reference sets. deduplication removing repeated text at URL, document, and line level; improves models, reduces memorisation, keeps evaluation honest. shingle an n-word window; a document is represented by its set of shingles. Jaccard similarity shared shingles ÷ union; the overlap measure near-duplicate detection targets. MinHash k hash minima per document; the fraction of matching minima estimates Jaccard; banded to make search linear. model-based quality filter a trained classifier scoring documents (reference-based, instruction-style positives, or LLM-rated); a previous model's opinion shaping the next model's data. decontamination removing documents overlapping evaluation benchmarks (e.g. 8-token spans); tiny in volume, decisive for trustworthy scores. data mixture / mixture weights the sources and the fraction of each batch drawn from each; chosen by small-model experiments, not by size. epochs per source how many times a pool is cycled; repetition is nearly free to ~4×, then decays. curriculum changing the mixture over the run: long-context stage near the end, annealing toward high-quality data during learning-rate decay. synthetic data model-written text used to fill gaps or supply verifiable supervision; kept a minority because of model collapse. model collapse successive generations trained on model output lose the distribution's tails. data wall the finite stock of public human text (a few hundred trillion tokens, a fraction usable) against runs already at fifteen trillion. SIMT / streaming multiprocessor (SM) a GPU's execution unit: many threads running one instruction on different data; an H100 has 132. tensor core hardware that multiplies a small matrix block in one instruction; the source of a GPU's headline FLOP/s. HBM the GPU's main memory (80–192 GB, 3–8 TB/s); where weights and activations live between operations. shared memory / SRAM ~228 KB per SM, several times HBM's bandwidth, beside the cores; where a kernel's working tile lives. NVLink / InfiniBand intra-node (~0.9 TB/s per GPU) and inter-node (~0.05 TB/s per port) links; each step outward is slower. memory bandwidth bytes per second between HBM and the cores; the "door". arithmetic intensity FLOPs performed per byte moved from HBM. ridge point peak FLOP/s ÷ bandwidth (H100 ≈ 295 FLOP/B); below it an op is memory-bound, above it compute-bound. roofline model achievable FLOP/s plotted against intensity: a diagonal (bandwidth) meeting a ceiling (peak). kernel fusion applying elementwise ops inside the matmul that produced their input, avoiding a second HBM pass. FlashAttention attention computed in SRAM tiles with an online softmax; memory linear in T, arithmetic unchanged, T×T never stored. online softmax running max m, sum ℓ, and output o, rescaled by e^(m_old − m_new) as tiles arrive; exact at the end. MFU (model FLOPs utilisation) 6·N·tokens/s divided by the cluster's peak; ≈ 40% for the best public frontier runs. mixture of experts (MoE) each block's MLP replaced by E expert MLPs; a router picks k per token; parameters held ≫ parameters run. router a small learned matrix scoring every expert for a token; softmax, top-k, renormalise. total vs active parameters stored (all experts) vs multiplied per token (k experts + attention); compute per token follows the active count. load balancing an auxiliary loss or a per-expert bias that keeps tokens spread across experts and GPUs. shared expert an expert every token uses, alongside the routed ones (DeepSeek). MQA / GQA one, or g groups of, key/value heads shared across query heads; KV cache shrinks by H or H/g. MLA (multi-head latent attention) store one compressed latent per token, reconstruct every head's K and V from it; cache ∝ latent dim, not head count. sliding-window attention attend only to the previous w tokens; cost and cache O(T·w) per layer; reach grows as ℓ·w with depth. local-global interleaving most layers windowed, a few full, so exact long-range lookup survives. attention sink the first token(s) kept permanently attendable so idle heads have a target. wavelength (RoPE) positions per full turn of pair i, 2π·base^(2i/d); pairs longer than the trained length never completed a turn. position interpolation / base scaling / YaRN squeeze positions; stretch only slow pairs; per-pair ramp plus a logit temperature fix. needle in a haystack a retrieval test at random depth in a long context; addressing a position is not the same as using it. data parallelism (DP) identical model copies, different batch slices; gradients averaged by all-reduce so all copies take the same update. all-reduce (ring) every GPU ends with the sum; each sends 2·(N−1)/N of the data regardless of N; hidden behind the backward pass by bucketing. reduce-scatter / all-gather the two halves of a ring all-reduce; ZeRO uses them separately. ZeRO stages 1–3 / FSDP shard optimiser state, then gradients, then weights across the DP group; per-GPU state falls toward 16·N/DP at ≈1.5× communication. tensor parallelism (TP) one matmul split across GPUs (columns of W_in, rows of W_out; heads for attention); one all-reduce of [T, C] per sub-layer per direction; kept inside the NVLink domain. pipeline parallelism (PP) consecutive blocks on different GPUs; one activation tensor crosses each stage boundary. pipeline bubble idle fraction (p−1)/(m+p−1) for p stages and m micro-batches; 1F1B bounds activation memory; interleaving shrinks it further. micro-batch / gradient accumulation the batch split into pieces whose gradients are summed before one optimiser step; mathematically identical to the large batch. context (sequence) parallelism (CP) one long sequence split by tokens across GPUs; K/V blocks passed round a ring for attention. expert parallelism (EP) MoE experts on different GPUs; tokens routed by all-to-all. 4D parallelism TP × PP × DP (× CP) whose product is the GPU count; Llama 3 405B: 8 × 16 × 128 on 16,384 H100s. mixed precision bf16 matmuls with an fp32 master copy for updates; fp16 needed loss scaling; FP8 with fine-grained scaling (DeepSeek-V3). activation checkpointing keep block inputs, recompute the rest during backward; ≈⅓ more compute for most of the activation memory. checkpoint (state) the saved 12–16 bytes/parameter training state; restart point after a failure. MFU model FLOP utilisation: 6N·tokens over peak FLOP/s × time; frontier runs 35–45%. straggler one slow device that stalls a synchronous step for all. hyperparameters the numbers fixed before a run: learning rate, batch size, weight decay, Adam betas, clipping, initialisation scale. warm-up the learning rate ramped from near zero over the first thousands of steps; protects the fragile early structure. stable phase the long middle of the run at (or gliding from) the peak learning rate. cosine schedule a smooth decay from peak to a small floor over a pre-set total length. warmup–stable–decay (WSD) hold the peak until late, then decay fast; lets a run be "finished" from any point of the stable phase. batch ramp small batches early (noisy gradients are still informative), large batches late (signal per token is small, cluster needs work). muP (maximal-update parametrisation) initialisation and per-layer learning-rate rules under which the best learning rate transfers across widths; how frontier rates are set from small sweeps. loss spike a sudden jump in training loss, often preceded by a gradient-norm spike; handled by rollback and skipping batches. QK-norm normalising queries and keys before the dot product so attention logits cannot drift and saturate softmax. z-loss a small penalty on the log softmax normaliser that keeps output logits centred. gradient clipping scale the gradient down when its global norm exceeds a threshold; insurance against freak batches. β₂ = 0.95 a fast-forgetting Adam second moment; the universal frontier choice for stability. checkpoint full training state (weights, optimiser, data position) written to storage; the rollback target; 16 bytes per parameter. annealing (cool-down) the final tokens with the learning rate decayed to zero on the highest-quality data; produces a visible final loss drop. long-context extension staged growth of the sequence length at the end of the run, with RoPE base adjusted and needle tests at each stage. effective training time the fraction of wall time spent training after failures and restarts; >90% on Llama 3 despite 466 interruptions. base model the pretrained checkpoint; continues text, does not answer. post-training the pipeline after pretraining: SFT → preference learning → RL → safety → evals, iterated; compute-light, labour-heavy. chat template the fixed way a conversation is flattened into one token stream with role markers. special tokens (chat) reserved ids (begin_of_text, start/end_header_id, eot_id in Llama 3) that mark turns and that the tokenizer never emits from user text. generation prompt the stream ending in an open assistant header; where the model's turn starts. end-of-turn token the special token the model samples to finish its turn; billed during SFT so the model learns to stop. supervised fine-tuning (SFT) Chapter 5's loop on conversations with a masked loss, small learning rate, few epochs. loss mask bill only assistant tokens (plus end-of-turn); everything else is context. rejection sampling generate many candidates from the current model, keep the best by a judge or verifier, use as SFT targets. catastrophic forgetting knowledge outside the fine-tuning slice decays when updates are too large; defended by small rates, few epochs, mixed data. LoRA freeze W, train A (C_in×r) and B (r×C_out), use W + A·B; r·(C_in+C_out) trainable numbers; mergeable after training. adapter any small trainable addition to frozen weights; LoRA is the common form. QLoRA LoRA over a 4-bit quantised frozen base. preference pair (x, y_w, y_l) a prompt with a chosen and a rejected response; the unit of preference data. Bradley–Terry model p(y_w ≻ y_l) = σ(r_w − r_l); pairwise votes become a score scale; fitted by −ln σ on pairs. reward model r_φ(x, y) a transformer with a scalar head trained on pairs with the Bradley–Terry loss; scores unseen responses like a rater would. RLAIF preference labels produced by a model judging against a rubric instead of by human raters. rejection sampling sample many responses, score them, keep the best; yields SFT data and pairs. policy π_θ the model being trained by RL or DPO. reference π_ref a frozen copy of the model at the start of the stage; the leash is measured against it. KL penalty (β) subtract β·(ln π_θ − ln π_ref) from the reward; keeps the policy where the reward model's scores mean something. RLHF sample from the policy, score with the reward model, subtract the KL term, update by policy gradient. PPO policy gradient with a clipped probability ratio so no step moves the policy far; needs a value model. value model / advantage a predictor of expected reward per prompt; the update is scaled by reward minus that expectation. reward hacking / over-optimisation the policy exploits errors in the learned reward; proxy score rises while true preference falls, past a KL that depends on reward-model quality. DPO direct preference optimisation: −ln σ(β·margin) where margin is the difference in log-probability shifts (vs the reference) between chosen and rejected; no reward model, no sampling. margin β·[(ln π_θ(y_w) − ln π_ref(y_w)) − (ln π_θ(y_l) − ln π_ref(y_l))]; the only thing the DPO loss depends on. on-policy vs off-policy training on samples from the model being trained vs from some other source; PPO is on-policy, DPO on a fixed set is off-policy, iterated DPO alternates. IPO / KTO / ORPO / SimPO the DPO family: squared target, single-response labels, no reference model, length-normalised reward. verifiable reward a reward computed by a program that checks the answer (equality, tests, a proof checker); exact and unlimited, but only for checkable tasks. verifier the checking program; hidden tests and sandboxes keep it honest. rollout / sample one full generation from the current policy at temperature > 0. policy / reference model the model being trained; the frozen copy it started from, used for the KL leash. group (G) several rollouts for the same prompt; the group's own statistics replace a value model. group-relative advantage (reward − group mean) / group std; same value for every token of a rollout. GRPO Group Relative Policy Optimisation: PPO's clipped ratio objective with group-relative advantages and no critic. probability ratio (ρ) new policy's token probability over the sampling policy's; clipped to 1 ± ε per step. KL penalty (β) the leash: penalises drift from the reference; β = 0 lets the policy collapse onto the reward. chain of thought / reasoning tokens intermediate text written before the answer; extra depth and external memory for a fixed-depth network. test-time compute accuracy bought after training by thinking longer or sampling more attempts; roughly log-linear in tokens. self-consistency (majority vote) sample n attempts, take the most common answer; needs wrong answers to disagree. best-of-n sample n, keep any that a verifier passes; 1 − (1−p)ⁿ with a perfect verifier. rejection sampling (for SFT) best-of-n used to make training data: keep verified traces, fine-tune on them (STaR). cold start a small SFT stage on clean reasoning traces before RL, for readability and speed. outcome vs process reward score the final answer vs score each step; the public frontier uses outcome rewards. overthinking / length hacking reasoning length inflating beyond need because length correlated with reward. distillation (of reasoning) fine-tuning a small model on a large reasoner's verified traces; beats RL on the small model directly. extended thinking (API) reasoning returned as a separate block, summarised; billed as output tokens. specification (constitution, model spec) the written statement of intended behaviour: helpful, honest, harmless, and how to trade them; text the model can read and be graded against. HHH helpful, honest, harmless; the three-way tension every spec resolves. alignment shaping θ so the model's own dispositions match the specification (narrow sense); matching what developers and users would want on reflection (broad sense). system-level safety classifiers, permissions, monitoring, and limits around the model that hold even when the model is wrong or fooled. constitutional AI the model critiques and revises its outputs against sampled principles; pairs become SFT and preference data. RLAIF reinforcement learning from AI feedback: preference labels produced by a model applying the spec, not by human raters. refusal / over-refusal / under-refusal declining a request; declining a benign one; answering a harmful one. One threshold, two error rates. jailbreak a user-crafted input that pushes the model out of its refusal region; can be social (framing) or mechanical (gradient-found suffix). prompt injection instructions arriving through retrieved or tool-returned content; indistinguishable from real instructions at the token level. defence in depth no single stage suffices; each assumes the others may fail. red-teaming attacking the model deliberately, by humans or by models, before users do. capability evaluations / scaling policy tests for dangerous capabilities and the published thresholds that gate deployment. sycophancy agreeing with the user against the evidence; a product of preference data amplified by RL. reward hacking / specification gaming optimising the proxy (length, agreement, tests pass) rather than the intent. alignment faking behaviour that matches the spec under observation and not otherwise; demonstrated in constructed settings. scalable oversight the open problem of judging outputs from models that exceed their judges. evaluation (eval) a task, a dataset, a metric, and a harness; the number is meaningless without all four. harness the code around an eval: prompt template, few-shot examples, decoding settings, answer parsing. benchmark a published eval with a fixed metric used to compare models; saturates within years. log-likelihood vs generative scoring score candidate answers by their probability (no generation) vs let the model write and parse the output. contamination benchmark items present in the training data; inflates scores; detected by held-out rewrites, mitigated by n-gram decontamination. prompt / format sensitivity score changes from relabelling options, reordering few-shot examples, or whitespace; report the spread and pair the harness. LLM-as-judge a model scoring model outputs against a rubric; near human-level agreement, with known biases. position bias / verbosity bias / self-preference the judge favours the first slot, the longer answer, or its own family; fixed by swapping, length control, and a different judge family. pairwise vs pointwise compare two answers, or score one on a scale; pairwise for decisions, pointwise for dashboards. arena anonymous human votes between two models' answers, turned into ratings. Bradley–Terry / Elo P(i beats j) = 1/(1 + e^{sⱼ−sᵢ}); fit strengths to votes; Elo is the online approximation. paired evaluation both models answer the same prompts; compare item by item; cancels the prompt set. sign test exact binomial test on decisive pairs under a fair-coin null; ties dropped. Wilson interval a confidence interval on a win rate that behaves at small n. sample size / power decisive pairs needed to detect an effect: ≈800 for 55%, ≈50 for 70%. decision evals vs release evals fast checkpoint checks during training vs full, decontaminated, published-harness evidence in a model card. evidence hierarchy paired human ratings > arena with intervals > calibrated judged pairs > decontaminated benchmarks with harness > bare scores > anecdote. decode step one pass that streams every weight once and produces one token per sequence in the batch; its time is set by bytes read, not arithmetic. continuous batching scheduling at the step level: finished sequences leave and waiting requests join at any step (Orca). paged KV cache / PagedAttention the cache cut into fixed blocks allocated on demand via per-sequence block tables (vLLM); removes reservation waste. prefix (prompt) caching identical prompt prefixes share cache blocks and skip prefill; priced lower by providers. speculative decoding a small draft model proposes k tokens; the target verifies all in one pass and accepts by a rule that preserves its distribution exactly. acceptance rate (α) per-token probability the target agrees with the draft; expected tokens per step = (1 − α^(k+1))/(1 − α). serve-time quantisation storing weights and cache in 8 or 4 bits to read fewer bytes per step and fit more cache; fp8 is the frontier default. disaggregated serving prefill and decode on separate GPU pools with the cache shipped between them. TTFT / TPOT time to first token (queue + prefill + one step) and time per output token (one step ÷ speculative gain). throughput tokens per second across all users; rises with batch until compute- or cache-bound. cost floor per token replica $/s divided by tokens/s; output tokens cost more because decode's batch is cache-capped. chunked prefill splitting a long prompt's prefill across several steps so it does not stall other users' decode. context (prompt) the single token sequence a request is flattened into: tools → system → turns, bracketed by special tokens; the only input you control. system prompt the operator section the model was post-trained to obey over the user; selects behaviour, cannot add knowledge. few-shot / in-context learning examples in the context that the model continues by attention pattern-completion (induction heads); format matters more than content. tool definition a JSON schema in the context describing a function your program runs; its description is a prompt. tool_use / tool_result the model's structured request block and your program's reply block; the loop runs until the model answers in text. structured output constrained decoding: illegal next tokens masked to −∞ at every step so the output matches a schema; guarantees form, not truth. strict tools the same masking applied to tool arguments. prompt caching the provider keeps the KV cache of a byte-identical prefix across requests; reads priced low; valid up to the first differing byte. cache breakpoint a marker (cache_control) after which caching stops; up to four per request; stable content before, volatile after. prompt injection instruction-shaped text in data slots (documents, tool results) that the model may obey; defended structurally, not by wording. thinking / effort provider-side controls on reasoning tokens before the answer; billed as output; pays on hard tasks only. context budget prefill, per-step attention, and lost-in-the-middle all grow with length; put in what the task needs, in reading order. retrieval-augmented generation (RAG) select a few relevant chunks from a private store and place them in the context; facts enter through the prompt, behaviour through training. chunk the unit of retrieval: a passage sized for matching (≈200–800 tokens), embedded separately from its document. retrieval embedding (bi-encoder) a transformer pooled to one vector per text, trained contrastively so questions land near the passages that answer them. contrastive loss softmax over similarities with the correct passage as target; negatives define what "different" means. cosine similarity dot product of unit vectors; the retrieval score for dense search. approximate nearest neighbour (ANN) an index (HNSW graph, IVF partitions) that finds near vectors in sub-linear time at a small cost in exactness. HNSW a layered graph of near neighbours searched by greedy hops from an entry point. product quantisation compressing vectors to a few bytes so billions fit in memory. vector database vectors plus an ANN index, chunk text, metadata filters, updates, and persistence. BM25 keyword scoring with inverse document frequency and length normalisation; catches exact strings dense search misses. hybrid search / reciprocal rank fusion run lexical and dense retrieval and merge by rank, 1/(60 + rank), so score scales need not match. reranker (cross-encoder) a model that reads query and chunk together and re-scores a shortlist; accurate, slow, second stage. query rewriting turning a conversational message into explicit search queries before retrieval. recall@k / MRR / nDCG retrieval metrics on labelled queries, measured before any answer is generated. lost in the middle accuracy drop when the relevant passage sits mid-context among many others. grounding instructing the model to answer only from retrieved excerpts and to cite them. agent a loop around model calls: the model emits text or a tool call; the host runs the tool, appends the result, and calls again until text or a budget. tool a function exposed to the model as text (name, description, JSON schema); the call is a sampled continuation, parsed by the API. tool_use / tool_result the blocks carrying a call and its result; all results of one turn go back in one user message. MCP (Model Context Protocol) an open standard for describing tools, resources, and prompts so one server works with any client. trajectory the full sequence of calls, results, and messages in one agent run; the unit of logging and evaluation. memory (context / scratchpad / long-term) the resent message list; a store the model writes and reads by choice; an external index retrieved from. compaction / context editing replacing older trajectory with a summary, or clearing old tool results, to keep long runs inside the window. error compounding task success ≈ pⁿ over n dependent steps; each step conditions on the last, mistakes included. looping re-issuing a call whose result is already in context; countered by budgets and repeated-call detection. budget a cap on steps, tokens, or dollars per task; the cheapest guardrail. approval gate a tool whose execution waits for a human; turns irreversible actions into review steps. permission scoping / sandbox the narrowest tool set and a bounded environment for whatever the agent can do. harness vs deployment who supplies the loop and context management, and who supplies the machine it runs on. case one prompt plus its context (fixture) and an expectation of what a passing answer does. eval set (golden set) the fixed collection of cases every variant runs on; grows as production failures become cases. expectation / rubric a statement of pass criteria specific enough that two people would agree on the verdict. fixture recorded tool readings handed to the application in place of live calls, so every variant sees the same world. both-directions coverage cases where the behaviour should happen and cases where it should not, so a lazy policy fails somewhere. grader / judge the function that turns an answer into a score; a judge is a model applying a rubric. null baseline a constant non-answer run through the whole pipeline; it must fail every case it should fail. judge calibration comparing judge verdicts with human labels; reported as sensitivity (passes caught) and specificity (failures caught). shrinkage factor a judge compresses every true difference by (sensitivity + specificity − 1). harness the code between cases and results; keeps errors, truncations, refusals, and wrong-model responses out of the scores. paired comparison comparing two variants case by case on the same cases. discordant / tied pairs cases where the variants disagree / agree; only discordant pairs carry evidence. sign test the probability that a win–loss split among discordant cases is at least this lopsided if the variants were equal. minimum detectable difference ≈ 2.8·√(discordant fraction / n); halving it needs four times the cases. outcome vs trajectory grading for agents: is the final result right, and was the path efficient and within guardrails. release gate thresholds written in advance (overall, per tag, regression, cost, p95 latency) that a candidate must all clear. pinned model id the exact model version requested, recorded alongside the model the response says served it. prompt version a hash computed from the system prompt and tool definitions, so any change produces a new version automatically. trace one structured, metadata-only record per request: status, steps, tools, token counts, cost, latency, versions. sampled transcript full request and answer text for a small fraction of traffic, kept apart, restricted, and short-lived. refusal rate / budget exhaustion / degraded rate the quiet failure signals of an LLM application; all return HTTP 200. drift quality falling while every request succeeds; caught by grading a daily sample of real traffic. degraded mode answering from a second model during a provider failure, flagged on the response and in the trace. server-side fallback an opt-in provider feature that re-runs a declined request on another model. exponential backoff with jitter retry delays that double each time and are randomised so clients do not retry in lockstep. shadow / canary / A/B rollout stages: run silently alongside, serve a small share and compare, run a deliberate experiment. canary gate a statistical comparison of canary against stable traffic that decides whether to widen the rollout. approval queue the service form of the agent's gate: proposals are recorded, a human approves and executes. data flywheel production failures become eval cases that gate the next change. interpretability reading a trained model's internals to explain its behaviour; observe, read out, intervene. linear probe logistic regression on activations for a property; success shows the property is present as a direction, not that it is used. activation patching overwrite one activation with its value from another run and re-run downstream; locates where an answer is carried and computed. ablation zero or mean-replace an activation to see what breaks; cruder than patching. attribution gradient of an output with respect to an internal activation; a cheap linear estimate of patching. circuit specific heads and units whose composition through the residual stream implements a behaviour (induction, IOI). superposition more features than dimensions, stored at angles; works because features are sparse. polysemantic neuron a coordinate that responds to several unrelated features; the symptom of superposition. feature a direction in activation space corresponding to an interpretable concept. sparse autoencoder (SAE) an overcomplete dictionary learned with a sparsity penalty; decoder rows are candidate features. steering / clamping adding or fixing a feature's activation during generation to change behaviour; an intervention that proves use. attribution graph a per-prompt graph of which features caused which, built from SAE-style features plus attribution. modality a kind of input or output: text, images, audio. patch a small square of an image (e.g. 16 × 16 pixels), flattened and projected into one vector; the image's token. Vision Transformer (ViT) a transformer over patch vectors; resolution sets sequence length. contrastive alignment (CLIP) training image and text encoders so that each image's softmax over a batch of captions picks its own caption. projector (adapter) a small learned map from encoder width to the language model's width. cross-attention attention whose queries come from one sequence (text) and keys and values from another (image). early fusion images tokenized into discrete ids in a shared vocabulary, read and generated by one transformer. image tokens image vectors in the token sequence; each costs what a word costs, including KV cache. spectrogram energy per frequency over short time windows; turns sound into an image-like grid. audio codec tokens discrete codes from a neural audio codec; let a transformer read and write sound. diffusion generation by repeatedly removing predicted noise from pure noise; trained by predicting added noise. latent diffusion diffusion run on a compressed autoencoder representation instead of pixels. classifier-free guidance moving each denoising step further toward the caption-conditioned prediction; trades variety for faithfulness. teacher / student the large model imitated and the smaller model trained to imitate it. distillation (logit) training the student to match the teacher's full next-token distribution, measured by KL divergence. distillation temperature dividing both models' logits by T > 1 to expose the teacher's ranking of unlikely tokens; loss scaled by T². sequence-level distillation fine-tuning the student on answers the teacher generated; works through an API. quantisation storing weights (or activations, or the KV cache) in fewer bits with a scale per group. group-wise scale one scale per 64–128 weights, so an outlier spoils only its own group. outlier features rare, very large activation dimensions that appear in large transformers and break naive quantisation. weight-only quantisation low-bit weights, 16-bit activations; captures the decode bandwidth saving. GPTQ / AWQ quantisation methods that compensate rounding error / protect weights that meet large activations. unstructured vs structured pruning zeroing individual weights (no speed-up without hardware support) vs removing neurons, heads, or blocks (real speed-up). depth pruning removing whole blocks; survivable because each block only adds to the residual stream. over-training training a small model on far more tokens than compute-optimal to cut lifetime inference cost. quality floor the minimum score on your own eval suite a model must reach before cost decides.