26  Multimodal Transformers

TipTL;DR

The Transformer wasn’t built for language — it was built for sequences of tokens. So any modality you can tokenise, it can model. ViT cuts an image into patch “tokens”; CLIP aligns image and text in a shared embedding space with a contrastive loss; modern vision-language models project image features into a frozen LLM’s token stream so one decoder reasons over both. The recipe is always the same: tokenise the modality, feed the same Transformer.

Depends on: BERT & GPT

26.1 Why this matters

Nothing in self-attention assumes words. It operates on a set of vectors with positions — pixels, audio frames, and protein residues all qualify. The multimodal era is the discovery that the architecture generalised for free: the hard part was never the model, it was turning each modality into tokens and deciding how to fuse them.

This is the natural endpoint of the era’s arc. The Transformer started as a translation model, became the universal language backbone, and is now the universal sequence backbone — one architecture spanning text, images, audio, video, and science.

26.2 The mechanism

26.2.1 Vision Transformer (ViT): Images as Tokens

ViT (Dosovitskiy et al., 2021) showed convolutions aren’t required for vision. Split an image into fixed patches (e.g. 16×16), flatten and linearly project each into a token embedding, add positional encodings, and feed a standard Transformer encoder. A [CLS] token aggregates for classification. Given enough data, ViT matches or beats CNNs — the same “scale beats inductive bias” lesson as in NLP.

26.2.2 CLIP: Aligning Two Modalities

CLIP (Radford et al., 2021) learns a shared embedding space for images and text without per-image labels. Two encoders (an image encoder, a text encoder) are trained with a contrastive objective on web image-caption pairs:

\[ \mathcal{L} = -\tfrac{1}{2}\big(\text{CE}_{\text{img}\to\text{txt}} + \text{CE}_{\text{txt}\to\text{img}}\big), \]

pulling matched image-text pairs together and pushing mismatched ones apart across a batch. The payoff is zero-shot classification: embed an image, embed candidate label strings (“a photo of a {class}”), pick the nearest. CLIP embeddings became the backbone of text-to-image generation and the visual front-end of many VLMs.

image patch tokens → image enc caption subword tokens → text enc shared embedding contrastive align matched pair → close mismatched → far Tokenise each modality, encode, and align in one space — the foundation of zero-shot vision and VLMs.

26.2.3 Vision-Language Models: One Decoder, Many Modalities

The dominant modern pattern keeps a pretrained LLM and grafts vision onto it:

  1. A vision encoder (often CLIP’s) turns an image into feature vectors.
  2. A small projector (an MLP, or cross-attention as in Flamingo’s (Alayrac et al., 2022) Perceiver) maps those features into the LLM’s embedding space — image “tokens”.
  3. The image tokens are interleaved with text tokens; the same decoder attends over both and generates text.

This is how GPT-4V, Gemini (Gemini Team, Google, 2023), LLaVA (Liu et al., 2023), and Qwen-VL work: vision becomes just more tokens in the context window. Generation in the other direction — text to image — runs through diffusion, increasingly with Diffusion Transformers (DiT) (Peebles & Xie, 2023) as the backbone, which powers image and video models like Sora.

WarningPitfall: fusion strategy is the real design choice

“Multimodal” hides a spectrum. Early fusion (tokenise everything into one stream, train jointly — Gemini) is most flexible but expensive. Late fusion (graft a vision encoder onto a frozen LLM via a projector — LLaVA) is cheap and fast to build but bottlenecked by the projector and the frozen weights. Most of a VLM’s behaviour — what it can and can’t perceive — is decided by where and how the modalities fuse, not by the Transformer itself.

TipGoing deeper
  • Subword tokenisation, the same idea ViT applies to patches — BERT & GPT.
  • Sora / DiT, Whisper, and other modality-specific models — Research Landscape.

26.2.4 Minimal sketch

Patchify an image into ViT tokens, and score CLIP-style image-text similarity.

import numpy as np

rng = np.random.default_rng(0)

def patchify(img, p):                          # img: (H, W) grayscale
    H, W = img.shape
    patches = [img[i:i+p, j:j+p].ravel()
               for i in range(0, H, p) for j in range(0, W, p)]
    return np.stack(patches)                    # (num_patches, p*p) = tokens

img = rng.normal(size=(8, 8))
tokens = patchify(img, 4)
print("patch tokens:", tokens.shape)            # (4, 16) -> linearly projected to d_model

def clip_score(img_emb, txt_embs):              # cosine similarity in shared space
    img_emb = img_emb / np.linalg.norm(img_emb)
    txt_embs = txt_embs / np.linalg.norm(txt_embs, axis=1, keepdims=True)
    return txt_embs @ img_emb

img_emb = rng.normal(size=8)
labels = rng.normal(size=(3, 8))                # "a photo of a {cat/dog/car}"
print("zero-shot logits:", np.round(clip_score(img_emb, labels), 2))
patch tokens: (4, 16)
zero-shot logits: [ 0.05 -0.4   0.34]

26.3 Application & impact

Model Move Lives on as
ViT image patches as tokens Vision backbone; encoder for VLMs
CLIP contrastive image-text alignment Zero-shot vision; text-to-image conditioning
Flamingo / LLaVA project vision into a frozen LLM The open VLM recipe
GPT-4V / Gemini native multimodal assistants Frontier multimodal products
DiT / Sora Transformer-backbone diffusion Image & video generation
  • The architecture generalised at no cost — the same block that read text now reads pixels, audio, and video, validating “it’s a sequence model” over “it’s a language model.”
  • CLIP’s shared space quietly underpins the generative-vision boom: it’s the conditioning signal for text-to-image diffusion.
  • The token stream became universal — the simplest path to a new modality is to tokenise it and feed the decoder you already have.
NoteKey takeaway

Multimodality is not a new architecture — it’s the Transformer applied to non-text tokens. ViT tokenises images, CLIP aligns modalities in one space, and VLMs project everything into a single decoder’s context: one block, attending over a sequence of tokens — only now the tokens can be anything. That same “tokenize anything” move is what lets the next two chapters push further — into generated pixels (Diffusion) and into physical actions (Embodied AI).

→ Next: Diffusion Transformers