Putting the right text in front of the model: embeddings, indexes, hybrid search, and how to know it worked.
Chapter 20 established that the only lever you hold at inference time is the context. This chapter is about filling it well. A frontier model knows nothing after its training cutoff, nothing about your company, and nothing about what happened this morning; and even a million-token window is not big enough to hold everything you might want it to consult on every call. Retrieval is the discipline of selecting, from a large private store, the few pieces of text that this particular question needs, and handing them to the model. The tools are the ones from Chapter 2, dot products between vectors, applied at the scale of millions of documents. Chapter 22 makes retrieval one tool among several that an agent can call.
The question this chapter answers: given a question and a million documents, how do you find the right few thousand tokens fast, and how do you measure whether you found them?
An on-call engineer at 3 a.m., pager buzzing. Somewhere in the team wiki, among four hundred pages, is the runbook that says exactly what to do. She does not read the wiki. She types three words into its search box, skims the top three results, opens one, and reads a paragraph.
Notice what made that work. The wiki was written before the incident; searching it costs seconds; a good search puts the right page in the top three; and she, not the search box, does the reading and the reasoning. If the search had returned the wrong page, she would have followed the wrong procedure with full confidence, because the page looked authoritative.
Retrieval-augmented generation is this scene with the model in the engineer's chair. The store is written ahead of time. Search is cheap and runs before every question. The model reads what search returned and reasons from it. And if search returns the wrong chunk, the model answers confidently from the wrong chunk. Most of the engineering is about that last sentence.
| In the picture | In the machine | The word we will use |
|---|---|---|
| The wiki | Documents split into passages, each stored with a vector and metadata | corpus, chunks, index |
| Three words in the search box | The question, turned into a vector by the same model that embedded the chunks | query embedding |
| How the search box ranks pages | Cosine similarity between query and chunk vectors, or word-overlap scoring, or both | dense retrieval, BM25, hybrid |
| Finding the top three without reading all four hundred | A graph or partition structure that finds near neighbours in sub-linear time | approximate nearest neighbour (ANN), vector database |
| Opening the best-looking result and reading closely | A second, more expensive model re-scores the shortlist | reranking (cross-encoder) |
| Pasting the paragraph into the ticket | Retrieved chunks placed in the context, with source labels | retrieval-augmented generation (RAG), grounding |
| "Did search find the right page?" | Scoring retrieval on labelled queries, independently of the answer | recall@k, MRR, nDCG |
Four reasons, each of which is a property of the machine you now know.
The alternative to retrieval is not "put it in the prompt anyway" but "fine-tune it in" (Chapter 14), and that is the wrong tool for facts: fine-tuning shapes behaviour reliably and stores facts unreliably, it cannot be updated per request, and it cannot be audited per answer. Retrieval is how facts enter; training is how behaviour enters.
Chapter 2 gave every token a vector and showed that dot products between vectors measure alignment. A retrieval embedding model does the same for a whole passage: text in, one vector out, typically 256 to 4,096 numbers, such that passages about the same thing point the same way. The model is usually a transformer of the kind you built in Part 1 with the unembedding replaced by a pooling step that averages or selects the final residual stream into one vector public.
What makes it a retrieval embedding is the training objective. Instead of next-token loss, the model is trained contrastively: given a query and the passage that answers it, pull their vectors together; given the same query and unrelated passages, push them apart. The loss is a softmax over similarities, the same softmax as Chapter 1, with the correct passage as the target:
one training example: query q, positive passage p⁺, negatives p₁⁻ … pₙ⁻ (often the other passages in the batch) scores sᵢ = cos(embed(q), embed(pᵢ)) / τ τ: a temperature, ≈ 0.02–0.05 loss −ln softmax(s)[p⁺] exactly the bill from Chapter 1, over passages instead of tokens gradient pushes embed(q) toward embed(p⁺) and away from every negative, in proportion to how much each was scored
Two consequences. First, the geometry is tuned for question-to-answer matching, not paraphrase: "why are iOS pushes failing" should land near a runbook paragraph that never contains the word "failing". Second, the negatives define what "different" means. Models trained with hard negatives (passages that look relevant but are not) discriminate finely; models trained only on random negatives learn topic but not specifics public.
The vectors are compared by cosine similarity, which is a dot product once the vectors are normalised to unit length. So a search over a million chunks is a matrix-vector product of shape [1M, D] × [D], one million dot products, a few milliseconds on a GPU. That is the brute-force baseline, and for corpora under a few hundred thousand chunks it is often all you need. Everything in §21.4 exists for when it is not.
Three passage vectors in a 4-dimensional embedding space (real ones have hundreds of dimensions; the arithmetic is identical), already unit length, and a query.
q = [ 0.70, 0.50, 0.10, 0.50 ] "queue growing, workers fine" p₁ (queue) = [ 0.80, 0.40, 0.00, 0.45 ] q·p₁ = 0.56 + 0.20 + 0.00 + 0.225 = 0.985 p₂ (crash) = [ 0.30, 0.85, 0.20, 0.40 ] q·p₂ = 0.21 + 0.425 + 0.02 + 0.20 = 0.855 p₃ (email) = [ 0.10, 0.10, 0.98, 0.10 ] q·p₃ = 0.07 + 0.05 + 0.098 + 0.05 = 0.268 ranking: p₁ (0.985) > p₂ (0.855) > p₃ (0.268)
The queue runbook wins by 0.13 over the crash-loop runbook. That margin is small, and it is typical: the top two candidates in real searches are often separated by a few hundredths, which is why a reranker (§21.5) that reads both passages closely earns its cost. The email runbook is far away, as it should be; the third coordinate it lives on is one the query barely touches.
A runbook is a document; a vector is one point. Embedding a whole ten-page document into one vector averages away the paragraph that answers the question. So documents are split into chunks, each embedded separately, and the chunk, not the document, is what gets retrieved. How to split is a real decision with real failure modes.
The trade-off has three corners. Small chunks (a sentence or two) give precise matches and cheap contexts, but the answer is often severed from the condition, table header, or definition that makes it correct. Large chunks (a page) keep context but dilute the vector, pull unrelated text into the prompt, and cost tokens on every question. Overlap between adjacent chunks softens the boundary problem at the cost of storing text twice. Published practice clusters around 200 to 800 tokens per chunk with 10 to 20 percent overlap, split at paragraph or heading boundaries rather than fixed word counts where the source has structure inferred from vendor guidance and RAG benchmark papers rather than a single canonical study.
Two refinements are worth knowing. Contextual chunking prepends a short description of the parent document to each chunk before embedding ("From the push-queue depth runbook: …"), so a chunk that says "scale from 12 to 24" also carries what it is about. Parent-child retrieval embeds small chunks for matching but returns the larger surrounding section for the context. Both attack the same problem: the best unit for matching is smaller than the best unit for reading.
One million dot products per query is fine. One billion is not, and neither is one million when you need ten thousand queries a second. A vector database is a store of vectors plus an index that answers "which stored vectors are nearest to this one" without touching them all. The answers are approximate: the index sometimes returns the second-nearest instead of the nearest. In exchange it is fifty to a thousand times faster public (ANN benchmark suites report these ranges across index types).
The index most systems use is HNSW, a hierarchical graph of near neighbours. Each stored vector is a node linked to a handful of its nearest neighbours. To search, start anywhere, look at the current node's neighbours, move to whichever is closest to the query, and repeat until no neighbour improves. Because each hop moves toward the query, a few dozen hops cross a million-node graph. A small upper layer of long-range links lets the first hops jump across the whole space before descending into the local neighbourhood.
The other family, IVF (inverted file), partitions the space into a few thousand cells by clustering, stores each vector in its cell, and at query time searches only the handful of cells nearest the query. It is simpler than HNSW and easier to keep on disk, at some cost in recall. Both are commonly combined with product quantisation, which compresses each vector to a few bytes so a billion of them fit in memory; the same trade of precision for size as weight quantisation in Chapter 27.
What a vector database adds beyond the index: storing the chunk text and metadata next to the vector, filtering by metadata before or during search (only this tenant's documents, only pages updated this year), incremental updates without rebuilding, and persistence. What it does not add: any understanding of the text. The index is a geometric structure over whatever vectors the embedding model produced. A bad embedding model indexed perfectly still retrieves badly.
Dense retrieval matches meaning and misses exact strings. Ask for "PayloadEncodeError" and a semantic embedding may return the general crash-loop runbook and the general error-handling page in either order, because to it the token is just a rare identifier. Ask a keyword index and it returns the one chunk containing that exact word instantly. The keyword method every search engine used before embeddings is BM25: score a chunk by how many query words it contains, weighted so that rare words count more (inverse document frequency) and long chunks do not win just by being long. It needs no model, no GPU, and no training, and it is embarrassingly hard to beat on queries with names, error codes, and identifiers public.
Hybrid search runs both and combines the rankings. The combination can weight the two scores, or, more robustly, use reciprocal rank fusion: each method contributes 1 / (60 + rank) for each chunk it ranked, and the sums are sorted. Fusion by rank sidesteps the fact that a BM25 score of 12.3 and a cosine of 0.83 are not on the same scale.
After the shortlist comes the reranker. A bi-encoder embeds the query and the chunk separately and compares vectors: fast, because chunk vectors are precomputed, but blind to interactions between the two texts. A cross-encoder feeds the query and the chunk together through a transformer and outputs a single relevance score: attention (Chapter 3) can now run between query tokens and chunk tokens, so it sees that "workers look fine" negates the crash-loop runbook. It is far more accurate and far too slow to run over a million chunks, so it runs over the top twenty to a hundred from the first stage. Two-stage retrieval, cheap-then-expensive, is the standard shape public.
One more lever before the index is touched: query rewriting. A user's message is rarely a good search query. "it's happening again, same as last week" retrieves nothing. A cheap model call can turn the conversation into two or three explicit queries ("push queue depth alert growing", "push worker restarts"), each run separately, results merged. Chapter 22's agents do this naturally by making retrieval a tool the model calls with a query it wrote.
A RAG system has two components that can fail: retrieval and generation. If you only measure final answers, you cannot tell which one failed, and you will tune the wrong one. So retrieval gets its own eval, and it is cheap to build: a list of questions, each labelled with the chunk or document that answers it, and a handful of metrics that need no model call.
k. The one that matters most, because a chunk outside the top k never reaches the model.1 / rank of the first correct chunk. Rewards putting it first, not just in the list.Here is the whole pipeline, chunking to metrics, in one script that runs without any API. The embedding is a signed hashing trick over IDF-weighted words: a crude bag-of-words vector, chosen so every step is visible. A real bi-encoder replaces one function.
$ .venv/bin/python code/ch21/rag_local.py10 docs → 21 chunks (60 words, overlap 15)
embedding matrix (21, 512), each row unit length
recall@k over 10 labelled queries
@1 @3
dense 0.80 1.00
bm25 0.90 1.00
hybrid 0.80 1.00
query: 'queue keeps growing but workers look fine'
0.0328 02-worker-crash-loop.md …# Runbook: push workers in a crash loop Symptoms: push-worker restart …
0.0323 01-push-queue-depth.md …# Runbook: push-queue depth alert Alert fires when the push queue dept…
0.0317 01-push-queue-depth.md …and needs no action. A growing queue means producers are outpacing the…
Read the last three lines. The top hit is the wrong runbook by a margin of 0.0005, because the crash-loop page mentions queue depth and workers too. The right page is second and third. At k = 3 the model would see both and could tell them apart; at k = 1 it would confidently explain how to quarantine a malformed payload to an engineer whose workers are fine. That is the failure mode of §21.7 reproduced on ten documents, and the fix is any of: a reranker that reads "workers look fine", a better embedding model, or simply a larger k with a model instructed to say which excerpt applies.
The failure modes are few and specific, and each maps onto a component.
| Symptom | Usually means | Fix lives in |
|---|---|---|
| Confident answer from the wrong document | Retrieval returned a near miss and the model trusted it | reranker, hybrid, higher k with instructions to choose, retrieval eval |
| "The documents don't mention this" when they do | Chunk boundary cut the answer from its context, or query phrasing did not match | chunking, contextual chunks, query rewriting |
| Right chunk retrieved, ignored in the answer | Lost in the middle: relevant chunk buried among many; long contexts attend unevenly public | fewer, better chunks; put the best first and last; reranking |
| Answers that were right last month are wrong now | Stale index: documents changed, vectors did not | incremental re-indexing, freshness metadata, version filters |
| Correct facts, wrong tenant | Metadata filter missing or applied after top-k | filter inside the index query, per-tenant namespaces |
| Citations to chunks that say something else | Model blended retrieved text with θ's prior | strict grounding instructions, quote-then-answer formats, citation checking |
Notice that half the table is retrieval and half is what the model does with what it was given. The second half is why grounding instructions matter: "answer only from the excerpts, cite the file, say if they do not cover it" turns a fluent guesser into a careful reader, and the API's citation features (Chapter 20) can make the citation a structured object the application can verify rather than a string it hopes is right.
With million-token windows, the question "why not just send everything?" is fair, and the answer is a calculation, not a principle.
corpus 400 runbook pages × 600 tokens = 240k tokens
questions per day 2,000
send everything 240k input tokens per question × 2,000 = 480M input tokens/day
at a frontier input price of a few $/M ≈ $1,000–2,500 per day, before caching
prefill latency for 240k tokens ≈ several seconds per question
plus: the model must find the paragraph among 240k tokens on every call
retrieve top-3 3 × 300 tokens + question ≈ 1k input tokens per question
2,000 × 1k = 2M input tokens/day, ≈ $5–10
plus: one vector search, ≈ milliseconds
prompt caching changes the first case: a cached 240k prefix costs a fraction to re-read (Ch 20)
but the cache is per exact prefix and must be rebuilt when any page changes
Retrieval wins by two to three orders of magnitude on cost and by seconds on latency when the corpus is large and questions are narrow. Long context wins when the corpus is small enough to cache, changes rarely, and questions need the whole thing at once (summarise this contract; find every inconsistency in this codebase). Many systems do both: retrieve a few thousand tokens of the most relevant material and also keep a cached, stable reference document in the prefix.
Dispatch gains a memory it does not have to carry in θ or in every prompt. The ten runbooks live in code/ch21/runbooks/; the local pipeline chunks, indexes, and searches them; the model gets the top three excerpts and strict instructions.
# code/ch21/dispatch_rag.py (excerpt)
import rag_local as R # chunks, embeddings, bm25, hybrid — built on import
import anthropic
client = anthropic.Anthropic()
SYSTEM = """You are Dispatch, the on-call assistant for Postbox. Answer using ONLY the runbook
excerpts provided. Cite the runbook file name in square brackets after each claim. If the
excerpts do not cover the question, say so."""
def answer(question, k=3):
hits = R.hybrid(question, k)
context = "\n\n".join(f"[{R.owners[i]}]\n{R.chunks[i]}" for i, _ in hits)
prompt = f"Runbook excerpts:\n\n{context}\n\nQuestion: {question}"
r = client.messages.create(model="claude-opus-5", max_tokens=1024, system=SYSTEM,
messages=[{"role": "user", "content": prompt}])
return "".join(b.text for b in r.content if b.type == "text"), [R.owners[i] for i, _ in hits](example output — retrieval is real and local; the model's text differs per run) Growing depth with healthy workers means producers are outpacing the pool [01-push-queue-depth.md]. Check the inbound event rate for a single-tenant spike, then scale the worker pool from 12 to 24 replicas with scale-workers; the change takes about 90 seconds [01-push-queue-depth.md] [07-scaling-workers.md]. Do not go above 48 replicas: the broker allows 50 connections and two are reserved for monitoring [07-scaling-workers.md]. Scale back down after the incident [07-scaling-workers.md]. retrieved: ['02-worker-crash-loop.md', '01-push-queue-depth.md', '07-scaling-workers.md']
The retrieved list includes the crash-loop runbook, the near miss from §21.6. With three excerpts and the grounding instruction, the model can see that the crash-loop page does not apply and answer from the other two, citing each. Had k been 1, the answer would have been about quarantining payloads. Retrieval quality and generation instructions are one system; the eval of Chapter 23 measures them together, and the retrieval eval of §21.6 measures the half you can fix cheaply.
| Quantity | Typical published values | Evidence |
|---|---|---|
| Embedding dimension | 256 – 4,096; 1,024 common | public (open embedding model cards) |
| Embedding model size | 100 M – 8 B parameters | public |
| ANN speed-up over brute force at ≈ 95% recall | 50 – 1,000× | public (ANN-Benchmarks) |
| Chunk size in practice | 200 – 800 tokens | inferred |
| Reranker shortlist | 20 – 100 candidates | inferred |
| How frontier assistants retrieve internally | Web search and file search exist as server-side tools; index internals are not disclosed | public for the tools, unknown for internals |
Embed whole documents, not chunks. One vector per runbook averages its five procedures into a blur. "iOS 403" lands near the APNs runbook but also near the FCM one, because both are "push failure documents"; the model receives ten pages and the answer is somewhere in them. Recall at the document level looks fine and the answers get worse, which is why retrieval evals label chunks.
Use dense retrieval only, on a corpus full of identifiers. Error codes, ticket numbers, hostnames, and function names are rare tokens with no learned meaning; a semantic embedding treats "PayloadEncodeError" and "InvalidProviderToken" as similar noise. Queries containing them retrieve by topic instead of by string, and the exact match that BM25 would have found instantly is missed. Hybrid search exists for this.
Skip the retrieval eval and tune on final answers. Every change to chunking, embedding, k, or the prompt moves the answer score, and you cannot tell which. Teams in this state tune prompts for weeks to compensate for a retriever returning the wrong chunk 30% of the time. Ten labelled queries and recall@k reveal it in an afternoon.
Retrieve fifty chunks "to be safe". Recall rises, and so does the token bill, the prefill latency, and the chance that the right chunk sits in the middle of forty-nine wrong ones and is under-attended. Published long-context tests show accuracy falling when the relevant passage is mid-context public. More is not safer past the point where the reranker has done its job.
Never re-index. Runbooks change; vectors do not. Six months later Dispatch confidently cites a procedure that was replaced, with a valid-looking citation to a file whose content has moved on. Freshness is metadata the index must carry and the pipeline must act on.
Say it back. The model cannot know what happened after its cutoff or what is in your private store, so facts enter through the context, and retrieval is how you pick which few thousand tokens to send. Documents are split into chunks sized for matching, and each chunk becomes a vector from an embedding model trained contrastively to pull questions toward the passages that answer them. A query becomes a vector the same way and is compared by cosine similarity, one dot product per chunk; for corpora too large or traffic too heavy for that, a graph or partition index finds near neighbours in sub-linear time at a small cost in exactness. Keyword search with BM25 catches the exact strings that meaning-vectors miss, and fusing the two rankings beats either. A cross-encoder reranker reads the shortlist closely and reorders it. The retrieved chunks go into the prompt with source labels and instructions to answer only from them. Retrieval is evaluated on its own with labelled queries and recall@k before anyone looks at answers, because its failures, wrong chunk, severed context, buried passage, stale index, are cheap to find there and expensive to find later. Long context is the alternative when the corpus is small and stable enough to cache; retrieval wins by orders of magnitude when it is not. Dispatch now reads its runbooks this way.
p₁ = [0.6, 0.8, 0], p₂ = [0, 0.6, 0.8], p₃ = [0.8, 0, 0.6] and a query q = [0.7, 0.7, 0.14] (normalise it first). Rank the three by cosine similarity. Then apply reciprocal rank fusion with a BM25 ranking of p₃, p₁, p₂ and give the fused order. Which passage moved, and why?rag_local.py, change the chunk size to 20 words with no overlap and re-run the recall table. Then add a fifth runbook query of your own whose answer straddles a chunk boundary and show its rank before and after restoring overlap. Finally, replace embed with a call to any embedding model you have access to and report the new recall@1.