15  The Transformer Era: A Map

TipTL;DR

Vaswani et al. (2017) replaces recurrence with self-attention and triggers three parallel lines of development: encoder-decoder models (T5, BART) for structured generation; encoder-only models (BERT, ViT) for understanding and representation; and decoder-only models (GPT family, LLaMA) that scale to become the dominant paradigm by 2022. Almost every modern system traces to one of these three branches.

Depends on: Additive and Multiplicative Attention

15.1 Three families from one paper

2017 2018 2019 2020 2021 2022 2023 2024 Enc-Dec Enc-only Dec-only Transformer Vaswani 2017 T5 Raffel 2019 BERT Devlin 2018 ViT Dosovitskiy 2020 CLIP Radford 2021 GPT-1 Radford 2018 GPT-2 Radford 2019 GPT-3 Brown 2020 InstructGPT Ouyang 2022 LLaMA Touvron 2023 o1 OpenAI 2024 Codex, … Encoder-Decoder Encoder-only Multimodal Decoder-only
Figure 1. Transformer family genealogy, 2017–2024. Three branches emerge from the original encoder-decoder Transformer (Vaswani 2017): encoder-only (BERT lineage, strong for understanding/retrieval), decoder-only (GPT lineage, now dominant at scale), and multimodal (CLIP, ViT — vision and cross-modal). Sparse MoE runs across all families as a scaling technique, not a separate branch.

15.2 Chapters in this section

Chapter What it covers
Self-Attention & Multi-Head Attention QKV projections, scaled dot-product, multi-head; the core mechanism
The Transformer Architecture Encoder block, decoder block, residuals, layer norm; full forward pass
Positional Encoding Sinusoidal, learned, RoPE, ALiBi — from fixed frequencies to rotation
BERT & GPT Encoder-only vs decoder-only pretraining; the fine-tuning paradigm
Output Layer & Decoding Logits, softmax/cross-entropy, weight tying; greedy, beam, top-k, nucleus
Scaling Laws & Emergence Kaplan 2020, Chinchilla — optimal compute allocation; capability jumps
Alignment & Instruction Tuning RLHF, InstructGPT, DPO — making large models follow instructions
Language Models — The Lineage n-grams → Bengio → RNN LMs → GPT→LLaMA→Gemini; the convergent recipe
Mixture of Experts Sparse routing, Switch Transformer, Mixtral — scaling compute efficiently
Beyond Quadratic Attention Mamba, RWKV, RetNet, hybrids — sub-linear alternatives to attention
Multimodal Transformers ViT + CLIP → GPT-4V, Gemini — unifying vision, language, audio

15.3 Key ideas to track across chapters

Three tensions run through the entire Transformer era; each chapter adds a layer:

  1. Parallelism vs. position. Self-attention sees all positions at once but has no built-in notion of order. Positional encoding is the fix, and every architectural generation has improved it.

  2. Pretraining objective. Masked LM (BERT) learns bidirectional representations; causal LM (GPT) learns to generate. Both are trained on the same data at the same scale but produce models with very different inductive biases.

  3. Scale vs. alignment. Scaling laws show that bigger models are predictably better at next-token prediction. But a better next-token predictor is not automatically a better assistant. Alignment techniques (RLHF, DPO) are the engineering bridge between raw capability and useful behaviour.

For a comprehensive reference organized by year, paradigm, and application area, see the Transformer Era Research Landscape.

→ Next: Self-Attention & Multi-Head Attention