Multimodal models

25.1An image becomes a sequence

A transformer needs a sequence of vectors. An image is a grid of pixels: a 224 × 224 colour image is 224 × 224 × 3 = 150,528 numbers. You could make each pixel a token, but at 50,176 tokens per small image, attention's T² (Chapter 3 §3.8) would be two and a half billion scores per head per layer. And a single pixel means almost nothing on its own, the way a single letter means almost nothing (Chapter 6).

The Vision Transformer, ViT, does the obvious thing and it works public (Dosovitskiy et al., 2020). Cut the image into a grid of 16 × 16-pixel squares. Flatten each square into one vector of 16 × 16 × 3 = 768 numbers. Multiply every one of them by the same learned matrix to get a vector of width C. Now the image is a sequence of 196 vectors, one per patch, in the same shape that word tokens have after the embedding lookup. Add positions, because attention cannot see order (Chapter 4 §4.4), and run the transformer blocks.

$ python code/ch25/patchify.py

image (224, 224, 3) → patches (196, 768) → tokens (196, 1024)
196 tokens of width 1024; each patch is 16×16×3 = 768 numbers before projection

Compare this with a text token. The text embedding is a lookup: token 791 selects row 791 of W_E. The patch embedding is a matrix multiply: the patch's pixels, 768 numbers, select nothing; they are projected. That difference matters. A text vocabulary is closed, 128k entries. The space of possible 16 × 16 patches is effectively infinite, so there can be no table, only a function. The patch projection is exactly the "matrix is a function" of Chapter 2 §2.4, applied to pixels.

Resolution is sequence length. Double the image's width and height and the patch count quadruples. A 1024 × 1024 image at 16-pixel patches is 4,096 tokens. That single fact explains most of the engineering in this chapter: every scheme for reading images is a trade-off between seeing fine detail (small text in a screenshot, a thin line on a chart) and paying for it in sequence length.

25.2Teaching pictures and words to agree

A ViT trained to classify images into 1,000 categories learns vectors that are good for telling cats from dogs. That is not what a language model needs. It needs the vector for an image of a dog on a beach to land near the vectors for "a dog on a beach", so that attention from text tokens can find what it is looking for (Chapter 3's query–key match), whatever words describe it.

CLIP learned exactly this, from 400 million image–caption pairs collected from the web public (Radford et al., 2021). Two encoders, one for images and one for text, each ending in a vector. Take a batch of N pairs. Compute the N × N matrix of similarities between every image vector and every caption vector (dot products of normalised vectors, cosine similarity from Chapter 2 §2.2). The diagonal holds the true pairs. Train both encoders so that each row's softmax puts its mass on the diagonal, and each column's too.

$ python code/ch25/contrastive_loss.py

similarity / tau (rows: images, cols: texts)
 [[ 6.46 -2.37  1.51 -3.97]
 [ 1.84  3.48 -4.29 -7.44]
 [-1.4  -1.12  7.32  1.58]
 [-5.08  1.79  2.24  9.07]]
p(text | image) rows
 [[0.99 0.   0.01 0.  ]
 [0.16 0.84 0.   0.  ]
 [0.   0.   1.   0.  ]
 [0.   0.   0.   1.  ]]
loss = 0.049   (uniform guessing would be ln 4 = 1.386)

Nobody labelled anything. The captions came with the images, written by the people who posted them. The batch supplies its own negatives: every other caption in the batch is a wrong answer. It is the same move that made next-token prediction scale (Chapter 1): a training signal that comes free with the data. The result is a shared space where text and images can be compared, and it is what most open vision–language models start from public.

25.3Three ways to wire vision into a language model

You have a language model that reads token vectors and a vision encoder that produces patch vectors. There are three published ways to connect them, and the difference is where the image enters the transformer.

A · Project into the sequence. Take the vision encoder's patch vectors, pass them through a small learned projector (a matrix, or a two-layer MLP) that maps encoder width to the language model's width C, and insert them into the token sequence where the image appeared. From the first block on, the language model treats them as tokens: self-attention reads them, the MLP transforms them, they occupy the KV cache. LLaVA showed that this works with a projector trained on a few hundred thousand image–text pairs and then a round of instruction tuning public (Liu et al., 2023). It is simple, it reuses a pretrained language model unchanged at first, and it is the dominant open design public.

B · Attend across. Keep the image vectors out of the sequence. Add cross-attention layers between some of the language model's blocks: the queries come from the text tokens (Chapter 3), the keys and values from the image vectors. Text can look at the image whenever it needs to, and the image costs nothing in the text sequence's length. Flamingo introduced this with gates that start closed, so the pretrained language model is undisturbed at the start of training public (Alayrac et al., 2022). Llama 3.2 Vision uses cross-attention adapter layers and keeps the text model's weights unchanged, so its text abilities are preserved public.

C · One vocabulary. Train an image tokenizer that turns an image into a short sequence of discrete ids from a learned codebook, the way BPE turns text into ids (Chapter 6). Add those ids to the vocabulary. Now a single transformer reads and writes both kinds of token, and generating an image is just generating tokens. Meta's Chameleon trained this way from scratch on mixed sequences public (2024). This "early fusion" is the design that most naturally produces images as well as reading them.

What do closed frontier models use? Their developers describe them as natively multimodal, trained on mixed text, image, and audio data from the start public (the claim itself). Which of these designs, or what mixture, is inside is not published unknown.

One detail completes design A. Chapter 4's RoPE encodes position along one axis, but an image has two. Qwen2-VL splits the rotary dimensions into groups for time, height, and width, so a patch's position in the grid is encoded in its rotations and a head can learn "the patch to the left of me" public. The same mechanism, one more axis.

25.4What an image costs

If image vectors enter the sequence, each one costs what a word costs: attention against every other token, a pass through every MLP, and a slot in the KV cache for the rest of the conversation (Chapter 7 §7.2). So the number of image tokens is the price, and every design chooses it differently:

• Fixed resize. LLaVA-1.5 resizes every image to 336 × 336 and uses 14-pixel patches: always 576 tokens public. Cheap and predictable, and fine print becomes unreadable.
• Tiles. Cut a large image into fixed-size tiles and encode each one, so detail survives up to a tile limit.
• Dynamic resolution. Qwen2-VL encodes the image at its own resolution with 14-pixel patches, then merges each 2 × 2 group of patch vectors into one token: a 1920 × 1080 screenshot is about 2,700 tokens public.
• The Claude API documents its count as roughly width × height ÷ 750 in pixels, with images whose long side exceeds 1,568 pixels scaled down first public (current rules in Appendix B). A 1920 × 1080 screenshot is scaled to 1568 × 882 and costs about 1,844 input tokens.

dashboard screenshot, 1920 × 1080     scaled to 1568 × 882 → 1,382,976 px ÷ 750   ≈  1,844 tokens
the same facts as text                "depth 12k→58k over 5 min; 12/12 workers
                                       healthy; inbound 1.9k→7.1k events/s"       ≈     40 tokens
a page of prose, 1,000 words                                                     ≈  1,300 tokens

in an agent loop (Chapter 24)         re-sent on each of 5 calls                  ≈  9,200 tokens of context

Dispatch reads a dashboard

Postbox engineers paste screenshots of dashboards into the incident channel. Dispatch can read them. The API accepts an image as a content block in the user message, next to the text:

# code/ch25/vision_call.py
with open("dashboard.png", "rb") as f:
    data = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": data}},
        {"type": "text", "text": "This is Postbox's push-queue dashboard. Is the queue draining or growing, and since when?"},
    ]}],
)

(example output — needs an API key and a screenshot; wording varies)
The queue is growing. Depth rises from about 12k at 02:35 to about 58k at 02:40, and the slope steepens
after 02:37. The worker-errors panel is flat, so this looks like more inbound traffic, not failing workers.
input tokens: 1869 | output tokens: 52

Put the image before the question, as the example does: the model reads the image tokens first and then the question can attend back to them, which is the order the causal mask of Chapter 3 favours. And treat what the model reads off a chart as a reading, not a measurement. Values read from pixels are approximate; when the dashboard has an API, Dispatch's tools from Chapter 20 are the better source for the exact numbers.

25.5Sound

Audio arrives as a waveform: 16,000 or more samples per second of air pressure. Like pixels, individual samples mean nothing and there are far too many of them. The standard first step turns sound into a picture. Slice the waveform into short overlapping windows, about 25 ms each, measure how much energy each window has at each frequency, and stack the windows side by side. The result is a spectrogram: time on one axis, frequency on the other, loudness as brightness. Speech in a spectrogram looks like a pattern of bands and strokes, and the tools of §25.1 apply to it directly.

Whisper, the best-documented speech model, turns 30-second windows of audio into 80-channel log-Mel spectrograms, encodes them with a transformer encoder, and has a transformer decoder produce the transcript one token at a time, attending across to the encoder's output public (Radford et al., 2022). That is design B of §25.3, applied to sound. It was trained on 680,000 hours of audio paired with transcripts from the web: the CLIP recipe, weakly labelled data at scale, applied to speech public.

Speaking back needs the reverse. Neural audio codecs compress a waveform into a short sequence of discrete codes, dozens per second, that a decoder can turn back into sound public (SoundStream, EnCodec). With such a codec, audio becomes tokens from a vocabulary, and design C applies: a transformer can read and write speech the way it reads and writes text. Frontier voice assistants that respond in speech directly, without a separate transcription step, are described by their developers as doing something of this kind inferred from published descriptions of end-to-end speech models; the exact codecs and designs are not published.

25.6Generating images

Reading an image turns pixels into vectors. Producing one runs the other way, and there are two families of method.

Tokens, one at a time. With an image tokenizer (design C), an image is a sequence of codebook ids, perhaps 1,024 of them for a 256 × 256 image. Generate them with the next-token loop of Chapter 7, then decode the ids back into pixels. It needs nothing new: the same sampler, the same KV cache, the same bill. Its weakness is the order. Images have no natural left-to-right, and one early mistake shapes every later patch, much as a model's early error propagates through a paragraph.

Diffusion: start from noise and remove it. The dominant method for high-quality images takes a completely different route public (Ho et al., 2020; Rombach et al., 2022). Training is simple to state. Take a real image, add a random amount of Gaussian noise, and train a network to predict the noise that was added, given the noisy image, the noise level, and the caption. The bill is squared error between predicted and actual noise, and the training signal again comes free with the data. Generation reverses it: start from pure noise, ask the network what noise it sees, subtract some of it, repeat a few dozen times. Each step makes the image slightly less noisy and slightly more like something matching the caption.

Two refinements make it practical. Latent diffusion runs the whole process not on pixels but on a compressed representation from an autoencoder, eight times smaller along each side, which cuts the cost by about sixty-fold and is what Stable Diffusion does public. Classifier-free guidance runs the denoiser twice per step, once with the caption and once without, and moves the image further in the direction of the difference; a guidance scale above 1 trades variety for faithfulness to the caption public (Ho and Salimans, 2021). The widget's guidance slider is that scale.

Current systems increasingly combine the two families: a transformer handles the text and the layout, and a diffusion-style decoder produces the pixels. Which combination closed frontier image generators use is not published unknown.

25.7Data and evaluation

A multimodal model is only as good as its pairing data. Three kinds are used, in order of the training stage where they matter public (LLaVA, Qwen-VL, and Llama 3 reports):

• Image–caption pairs from the web, billions of them after filtering (LAION-5B is a public example). Noisy, cheap, and good for alignment: they teach that this picture goes with these words.
• Interleaved documents: web pages with images in place among the text. They teach that images appear in the middle of arguments and are referred to by the surrounding words.
• Instruction data: questions about images with good answers, much of it synthesised by a strong model from captions and then filtered. Charts, documents, screenshots, and diagrams are over-represented on purpose, because they are what users ask about and what captions rarely describe.

The same filtering questions as Chapter 9 apply, with two additions: whether the caption actually describes the image (a large fraction of web alt-text does not), and whether images of real people, documents, and screens raise privacy problems that text data does not.

Evaluation follows Chapter 18's pattern, with benchmarks for each kind of looking: general visual questions, college-level questions with diagrams (MMMU), reading documents and forms (DocVQA), reading charts (ChartQA) public. Contamination is harder to check than for text, because the same image appears on the web at many sizes and crops. And for applications, Chapter 23 applies unchanged: if Dispatch reads dashboards, the eval set needs dashboard screenshots with expectations a person has checked, including ones where the right answer is "the chart does not show that".

25.8Beacon's numbers