46  Transformer Era: Research Landscape

← Back to: The Transformer Era — A Map

TipAbout this page

A structured reference map of the Transformer era (2017 — present), organized three ways: chronological, by architecture family, and by application area. Each entry states the primary contribution, the key formalization where one exists, and why it mattered. Use it as a reading guide or as a lookup when a chapter references a work without full derivation.

Years reflect first public availability (arXiv or blog post). Citation links point to the peer-reviewed venue, which may differ by up to one year.

46.1 Chronological timeline

Year Work Family Notes
2017 Sparse MoE
Shazeer et al.		 Arch
  • Token-level routing: each token activates top-k of N expert FFNs; rest are skipped
  • y = Σ G(x)_i · E_i(x) where G is a sparse gate
  • First demonstration that MoE can scale Transformers without proportional compute; foundation of GPT-4, Gemini, DeepSeek
2017 Transformer [@vaswani2017]
Vaswani et al.		 Enc-Dec
  • Self-attention replaces recurrence entirely; full encoder-decoder for MT
  • Attn(Q,K,V) = softmax(QKᵀ/√d_k)V
  • Foundation of every model in this table; 2017 WMT SOTA
2018 GPT-1 [@radford2018gpt]
Radford et al. (OpenAI)		 Dec-only
  • Decoder-only causal LM pretrained on BooksCorpus; fine-tuned per task
  • Causal mask: upper-triangular -∞ prevents attending to future tokens
  • Established pretraining + fine-tuning for generation; direct ancestor of GPT-4
2018 BERT [@devlin2019]
Devlin et al. (Google)		 Enc-only
  • Masked LM + Next Sentence Prediction; bidirectional representations
  • 15% of tokens replaced with [MASK]; model predicts them from full context
  • Dominated GLUE / SQuAD; defined the encoder fine-tuning paradigm used through 2022
2019 GPT-2 [@radford2019gpt2]
Radford et al. (OpenAI)		 Dec-only
  • Scale to 1.5B params on WebText; zero-shot task transfer without any fine-tuning
  • Pre-LN architecture; larger BPE vocabulary (50 257 tokens)
  • First convincingly human-quality prose; sparked public AI safety debate
2019 RoBERTa [@liu2019roberta]
Liu et al. (Facebook)		 Enc-only
  • BERT replication with better training: no NSP, dynamic masking, longer runs
  • No architecture change — training recipe is the difference
  • Set a new BERT baseline; showed that pretraining details dominate architecture choice
2019 T5 [@raffel2020]
Raffel et al. (Google)		 Enc-Dec
  • Every NLP task recast as text-to-text: input and output are always strings
  • Unified format: "translate English to German: ..." → target string
  • Systematic scaling study on C4; text-to-text framing adopted by FLAN, mT5, UL2
2019 BART [@lewis2019bart]
Lewis et al. (Facebook)		 Enc-Dec
  • Denoising pretraining: corrupt input text (masking, deletion, permutation), train encoder-decoder to reconstruct
  • Corruption functions include token masking, text infilling, document rotation, sentence permutation
  • SOTA on CNN/DailyMail summarisation; strong baseline for conditional generation; widely used in multilingual settings
2020 Scaling laws [@kaplan2020]
Kaplan et al. (OpenAI)		 Research
  • Loss follows power laws in N (params), D (tokens), C (compute) independently
  • L(N) ∝ N^{-α_N}; optimal allocation: N ∝ C^{0.73}
  • Changed how every subsequent model was budgeted; predicted GPT-3 performance before training
2020 GPT-3 [@brown2020]
Brown et al. (OpenAI)		 Dec-only
  • 175B params; in-context few-shot learning without any weight updates
  • Prompt: [example₁] [example₂] [query] → completion (no gradient)
  • Emergence of reasoning at scale; API-as-product model; opened the LLM era
2020 ViT [@dosovitskiy2021]
Dosovitskiy et al. (Google)		 Enc-only
  • Images split into 16×16 pixel patches; each patch linearly projected and treated as a token
  • z₀ = [x_cls; x¹E; …; x^N E] + E_pos
  • Proved Transformers competitive with CNNs at scale; spawned DeiT, Swin, DINO, SAM
2021 Switch Transformer [@fedus2021switch]
Fedus et al. (Google)		 Arch
  • Simplifies MoE to top-1 routing (one expert per token); billion-to-trillion params with controlled compute
  • Auxiliary load-balancing loss: L_aux = α · Σ f_i · p_i
  • Demonstrated trillion-parameter models are trainable; foundation of Mixtral and DeepSeek MoE
2021 CLIP [@radford2021clip]
Radford et al. (OpenAI)		 Multimodal
  • Contrastive pretraining on 400M image-text pairs; separate image and text encoders
  • Maximize cos(f_I, f_T) for matching pairs; InfoNCE loss
  • Universal vision-language backbone; powers DALL-E 2, Stable Diffusion conditioning
2021 AlphaFold 2 [@jumper2021]
Jumper et al. (DeepMind)		 Science
  • Evoformer applies attention over multiple sequence alignments; geometry-aware pair representations
  • Triangle attention: a_{ij} = softmax(b_{ij} + Σ_k b_{ik} + b_{kj})
  • Solved 50-year protein structure prediction problem; directly accelerated drug discovery
2021 Codex [@chen2021codex]
Chen et al. (OpenAI)		 Dec-only
  • GPT fine-tuned on 54M public GitHub repos; generates, completes, and explains code
  • pass@k metric: probability at least 1 of k samples passes all unit tests
  • Ships as GitHub Copilot (2022); established code as a first-class LLM domain
2022 Chinchilla [@hoffmann2022]
Hoffmann et al. (DeepMind)		 Research
  • Kaplan exponents were wrong: params and tokens should scale equally for compute-optimal training
  • N* ≈ D* ≈ (C / 6)^{0.5}; GPT-3 was under-trained by ~4×
  • Invalidated the GPT-3 era strategy; every post-2022 model (LLaMA, Gemini, Mistral) follows this
2022 LoRA [@hu2022lora]
Hu et al. (Microsoft)		 PEFT
  • Freeze pretrained weights; inject low-rank decomposition ΔW = BA (rank r ≪ d)
  • W' = W₀ + BA, only A,B trained; reduces trainable params by 10 000×
  • Made fine-tuning accessible on consumer hardware; used in almost every open fine-tune
2022 Chain-of-Thought [@wei2022cot]
Wei et al. (Google)		 Prompting
  • Adding intermediate reasoning steps to few-shot examples dramatically improves multi-step reasoning
  • Prompt: "Let's think step by step..." before the answer
  • Revealed emergent reasoning in large models; direct precursor to o1 and test-time scaling
2022 Flash Attention [@dao2022flashattn]
Dao et al. (Stanford)		 Efficiency
  • IO-aware exact attention: tile softmax computation, fuse kernels, avoid materialising O(N²) attention matrix in HBM
  • Online softmax; recompute activations in backward pass rather than store
  • 2–4× speedup; enabled contexts beyond 8K; universally adopted from 2023 onward
2022 InstructGPT [@ouyang2022]
Ouyang et al. (OpenAI)		 Alignment
  • RLHF pipeline: supervised fine-tuning → reward model → PPO policy optimisation
  • r(x,y) − β log[π_RL(y|x) / π_SFT(y|x)]
  • Direct precursor to ChatGPT; established RLHF as the default alignment method
2022 Whisper [@radford2022whisper]
Radford et al. (OpenAI)		 Enc-Dec
  • Encoder-decoder Transformer trained on 680K hours of web audio in 96 languages
  • Mel spectrogram encoder → text decoder; multitask via special tokens (<|translate|>, etc.)
  • First robust open-weights ASR; standard baseline; Whisper large v3 still used in 2026
2022 ChatGPT
OpenAI		 Dec-only
  • GPT-3.5 + RLHF + instruction tuning deployed as a conversational product
  • 100M users in 2 months; redefined public understanding of AI
2023 QLoRA [@dettmers2023qlora]
Dettmers et al. (UW)		 PEFT
  • Quantise frozen base model to 4-bit NF4; inject trainable LoRA adapters; backpropagate through quantised weights
  • NF4 format: 4-bit normal float quantisation preserving gradient signal; paged optimisers for memory spikes
  • Fine-tune a 65B model on a single 48 GB GPU; democratised instruction tuning; basis of Guanaco and most open fine-tunes
2023 Speculative decoding [@chen2023speculative]
Chen et al. (Google)		 Efficiency
  • Small draft model proposes k tokens; target model verifies all in a single forward pass; accept or reject via sampling
  • Acceptance criterion preserves exact target distribution: p_accept = min(1, p_target / p_draft)
  • 2–3× inference speedup with zero quality loss; now standard in production (TGI, vLLM, TensorRT-LLM)
2023 LLaMA [@touvron2023llama]
Touvron et al. (Meta)		 Dec-only
  • Open weights with Chinchilla-optimal training; modern SLM stack: RoPE + RMSNorm + SwiGLU + GQA
  • RoPE: q_m · k_n → Re[(q_m e^{imθ})(k_n e^{inθ})^*]
  • Baseline for all open-source research; spawned Alpaca, Vicuna, Mistral, Llama 2/3
2023 GPT-4
OpenAI		 Dec-only
  • Multimodal (image + text input); large-scale RLHF; architecture undisclosed but widely assumed MoE
  • Bar exam top-10%, medical licensing, coding competitions — benchmark SOTA across domains
2023 DPO [@rafailov2023]
Rafailov et al. (Stanford)		 Alignment
  • Alignment without RL: directly optimises a closed-form loss derived from the RLHF reward
  • L_DPO = −log σ[β log(π(y_w)/π_ref) − β log(π(y_l)/π_ref)]
  • Simpler and more stable than PPO; default fine-tuning method for most open-source models
2023 Mistral 7B [@jiang2023mistral]
Jiang et al. (Mistral AI)		 Dec-only
  • GQA + sliding window attention; outperforms LLaMA-2 13B on most benchmarks at 7B params
  • GQA: fewer KV heads than Q heads; sliding window: attend only to last w tokens per layer
  • Demonstrated architecture efficiency matters at 7B; led to Mixtral (sparse MoE, Dec 2023)
2023 Mamba [@gu2023mamba]
Gu & Dao (CMU / Stanford)		 Alt-Arch
  • Selective state space model: input-dependent SSM parameters let the model filter or remember content selectively
  • Hidden state: h_t = A_t h_{t-1} + B_t x_t; output: y_t = C_t h_t — A,B,C are functions of input
  • Linear-time inference; matches Transformer quality at language modelling; opened serious competition to quadratic attention
2023 Gemini [@team2023gemini]
Google DeepMind		 Multimodal
  • Natively multimodal decoder (text, image, audio, video, code); three sizes (Ultra, Pro, Nano); MoE-based
  • Trained on interleaved multimodal sequences end-to-end rather than late-fusion of separate encoders
  • First model to match or exceed GPT-4 on key benchmarks at release; Gemini 1.5 introduced 1M-token context window
2024 o1
OpenAI		 Dec-only
  • Test-time compute scaling: extended chain-of-thought as inference, not training
  • RL on verifiable reasoning tasks; more inference tokens → better answers
  • Opened a second scaling axis (inference FLOPs); reasoning model paradigm now widespread
2024 Sora
OpenAI		 Video
  • Diffusion Transformer (DiT): patches spacetime video into tokens; Transformer denoising backbone replaces UNet
  • Unified representation of images, video, audio as sequences of spacetime patches at variable resolution and length
  • First model to generate coherent minute-long videos with object permanence; "GPT-1 moment" for video generation
2024 DeepSeek-V2 / MLA [@deepseekv2team2024]
DeepSeek AI		 Efficiency
  • Multi-head Latent Attention (MLA): compress K,V into a low-rank latent vector c_t^{KV}, reconstruct via up-projection at inference
  • KV cache per token: d_c ≪ n_h × d_h (e.g., 5× compression vs standard MHA)
  • Reduces KV cache by up to 93%; enables longer context at lower memory cost; adopted in DeepSeek-V3/R1
2025 DeepSeek-R1
DeepSeek AI		 Dec-only
  • Open reasoning model trained with RL only (GRPO — no value model); 671B sparse MoE
  • GRPO: group relative policy optimisation using outcome rewards only
  • o1-level reasoning at ~1/30 cost; largest open-weights frontier model as of early 2026

46.2 By architecture family

Encoder-Decoder

Two stacks: encoder reads the full source bidirectionally; decoder generates target tokens autoregressively with cross-attention to all encoder states. The original Transformer family, and still dominant for structured generation (MT, summarisation, speech).

Work Year Key specialisation
Transformer (Vaswani et al., 2017) 2017 Machine translation; established the architecture
T5 (Raffel et al., 2020) 2019 Text-to-text unification; systematic scaling study
BART (Lewis et al., 2019) 2019 Denoising pretraining; strong summarisation
Whisper (Radford et al., 2023) 2022 Multilingual speech; weak-supervision at scale
mT5 (Xue et al., 2021) 2021 Multilingual T5 across 101 languages

Encoder-decoder peaked around 2020–2022. Decoder-only models at sufficient scale can perform the same tasks, reducing incentive to maintain the split.

Encoder-only

A bidirectional Transformer pretrained on a self-supervised objective; fine-tuned representations for downstream tasks. Strong for classification, extraction, and retrieval; cannot generate fluently.

Work Year Objective Key strength
BERT (Devlin et al., 2019) 2018 MLM + NSP Bidirectional; defined fine-tuning paradigm
RoBERTa (Liu et al., 2019) 2019 MLM only Better training recipe; new BERT baseline
ViT (Dosovitskiy et al., 2021) 2020 Patch classification Transformers for vision
CLIP (Radford et al., 2021) 2021 Contrastive image-text Universal multimodal backbone
DeBERTa (He et al., 2021) 2021 MLM + disentangled attn Disentangled position representations

Now used primarily for embeddings, retrieval (dense passage retrieval, bi-encoders), and cross-modal alignment rather than generation.

Decoder-only

A causal (left-to-right) Transformer pretrained as a language model. By 2022 this family dominates every benchmark and every major product.

Work Year Scale Key advance
GPT-1 (Radford et al., 2018) 2018 117M Pretraining + fine-tuning paradigm
GPT-2 (Radford et al., 2019) 2019 1.5B Zero-shot task transfer
GPT-3 (Brown et al., 2020) 2020 175B In-context few-shot learning; emergence
Codex (M. Chen et al., 2021) 2021 12B Code generation; GitHub Copilot
InstructGPT (Ouyang et al., 2022) 2022 175B RLHF alignment
LLaMA (Touvron et al., 2023) 2023 7–65B Open weights; modern SLM architecture stack
GPT-4 2023 undisclosed Multimodal; SOTA across all benchmarks
o1 2024 undisclosed Test-time compute scaling
DeepSeek-R1 2025 671B (MoE) Open frontier reasoning model

Mixture of Experts (MoE)

A routing mechanism layered on top of any Transformer: each token activates only a subset of FFN experts, keeping active compute constant while total parameters grow.

Work Year Routing Notable use
Sparse MoE (Shazeer et al., 2017) 2017 Top-k soft gate Founding paper; 137B params
Switch Transformer (Fedus et al., 2022) 2021 Top-1 hard Trillion-param training
Mixtral 8×7B (Jiang et al., 2024) 2024 Top-2 of 8 First competitive open MoE
DeepSeek-V3 (2024) 2024 Top-2 of 256 671B sparse; frontier quality at open cost

MoE is the dominant architecture at frontier scale. GPT-4, Gemini 1.5, and almost every post-2023 flagship model is believed to use sparse MoE.

Methods: training, alignment, efficiency

Works that changed how models are trained, aligned, or deployed without introducing a new architecture family.

Work Year Type Core contribution
Scaling laws (Kaplan et al., 2020) 2020 Training Power-law training budgets
Chinchilla (Hoffmann et al., 2022) 2022 Training Optimal token/param ratio; equal scaling
LoRA (Hu et al., 2022) 2022 PEFT Low-rank weight injection; 10 000× fewer trainable params
Chain-of-Thought (Wei et al., 2022) 2022 Prompting Intermediate reasoning steps unlock multi-step tasks
Flash Attention (Dao et al., 2022) 2022 Efficiency IO-aware exact attention; 2–4× speedup
InstructGPT RLHF (Ouyang et al., 2022) 2022 Alignment PPO from human preferences
DPO (Rafailov et al., 2023) 2023 Alignment Closed-form preference loss; replaces PPO
RAG (Lewis et al., 2020) 2020 Retrieval Non-parametric memory: retrieve passages, condition generation on them
RoPE (Su et al., 2024) 2021 PE Rotary position encoding; adopted by LLaMA family
MQA (Shazeer, 2019) 2019 Efficiency Multi-query attention: all Q heads share one K,V head
GQA (Ainslie et al., 2023) 2023 Efficiency Grouped-query: G shared KV groups; interpolates MHA↔︎MQA
MLA (DeepSeek-AI, 2024) 2024 Efficiency Low-rank KV compression; up to 93% KV cache reduction
QLoRA (Dettmers et al., 2023) 2023 PEFT 4-bit NF4 quantisation + LoRA adapters; single-GPU fine-tuning
Speculative decoding (C. Chen et al., 2023) 2023 Efficiency Draft-verify pipeline; 2–3× throughput, exact distribution

Alternative architectures: beyond quadratic attention

The Transformer’s \(O(N^2)\) attention cost motivates research into sub-linear alternatives. None has displaced Transformers at scale, but Mamba and hybrids are serious contenders for 2025–2026.

Work Year Mechanism Status
Mamba (Gu & Dao, 2023) 2023 Selective SSM — input-dependent state transitions Competitive at small scale; Mamba-2 unifies SSM ↔︎ attention
RWKV (Peng et al., 2023) 2023 Linear attention via token-shift recurrence LLM-scale open model (14B); constant inference memory
RetNet (Sun et al., 2023) 2023 Retention: parallel train, sequential infer “Training Parallelism + Inference Efficiency + Performance” triangle
Griffin (DeepMind, 2024) 2024 Gated linear recurrence + local attention hybrid Matches Llama at 14B; inference-efficient

The most likely outcome is hybrid architectures that mix attention layers (for long-range global context) with SSM or linear recurrence layers (for local computation efficiency).

NoteOut of scope: Graph Neural Networks

Graph Neural Networks (GCN, GAT, and successors) address a different input structure, arbitrary graphs rather than sequences or grids, and are deliberately out of scope for this timeline. They are omitted by design, not by oversight: this book traces the sequence-and-grid lineage (RNN → CNN → Transformer) rather than the parallel graph-representation-learning lineage.

46.3 By application area

Domain Landmark work Available Current state
Machine translation Transformer 2017 2017 Transformer MT is the industry standard
NLU / QA / classification BERT 2018 2019 Fine-tuned encoders or few-shot LLMs
Text generation GPT-2 2019 2019 GPT-4o, Claude 4, Gemini 2.x, LLaMA 3
Code generation Codex 2021 2022 (Copilot) Claude, DeepSeek-Coder, GPT-4o
Image classification ViT 2020 2021 ViT, DeiT, Swin, DINO v2
Image generation CLIP + diffusion 2021 2022 Stable Diffusion, DALL-E 3, Flux, Midjourney
Video generation Sora (DiT) 2024 2024 Sora 2, Veo 2, Kling, Wan
Speech recognition Whisper 2022 2022 Whisper v3, Distil-Whisper
Protein structure AlphaFold 2 (Jumper et al., 2021) 2021 2021 AlphaFold 3, ESMFold, ESM3
Reasoning CoT (Wei et al., 2022) + o1 2024 2024 o3, DeepSeek-R1, Gemini Thinking
Agents and tool use GPT-4 function calls 2023 2023 Multi-agent frameworks, Claude tool use
Long-context understanding Flash Attn + LLaMA 2023 2023 1M+ context windows (Gemini 1.5 Pro)
Efficient fine-tuning LoRA 2022 2022 QLoRA, DoRA, widely used open-source

46.4 Reading order

Core architecture sequence (read the chapters in this order):

  1. Self-Attention & Multi-Head Attention
  2. The Transformer Architecture
  3. Positional Encoding
  4. BERT & GPT
  5. Output Layer & Decoding
  6. Scaling Laws & Emergence
  7. Alignment & Instruction Tuning
  8. Language Models — The Lineage
  9. Mixture of Experts
  10. Beyond Quadratic Attention — SSMs and Hybrids
  11. Multimodal Transformers

If you want the deployment angle first: start with Scaling Laws → Alignment → MoE, then work backward through the architecture chapters.