23  Language Models: The Lineage

TipTL;DR

“Language model” is older than the Transformer by half a century: a language model is anything that assigns probability to the next token, \(p(x_t \mid x_{<t})\). The idea runs from n-gram counting (Shannon) through the neural probabilistic LM (Bengio 2003) and recurrent LMs (Mikolov 2010) to the decoder-only Transformer line that dominates today: GPT-2 → GPT-3 → GPT-4, branching into PaLM, LLaMA, Mistral, DeepSeek, Qwen, Gemini. This chapter is the lineage — the long history first, then how the era’s ingredients assembled into modern LLMs. The Research Landscape is the full catalog.

Depends on: Scaling Laws & Emergence · Alignment & Instruction Tuning

23.1 Why this matters

By now the core of a modern LLM has been built piece by piece — attention, the block, position, pre-training, decoding, scaling, alignment. But two things are still missing. First, the pieces have never been assembled into the story of how we got ChatGPT, LLaMA, or Gemini. Second, the chapters jumped straight to the Transformer, skipping the fifty years of language modelling that came before — the history that explains why next-token prediction is the objective at all.

This chapter supplies both: a concise history of language models, then the modern lineage. It is deliberately conceptual — it tracks families and their contributions, not benchmark numbers or parameter counts (those go stale within months and live in the landscape).

23.2 A brief history of language models

Long before neural networks, a language model was already defined the way it is today: a distribution over the next symbol given the previous ones, \(p(x_t \mid x_{<t})\). What changed across eras is how that distribution is estimated.

Statistical n-grams + smoothing Shannon 1948– Neural distributed word reps Bengio 2003 Recurrent RNN / LSTM LMs Mikolov 2010 Transformer GPT line, scale 2017– Same objective — p(next | context) — four ways to estimate it Each era kept the goal and changed the estimator: counts → a feedforward net → recurrence → attention at scale.

Statistical / n-gram era (Shannon onward). The earliest LMs estimate \(p(x_t \mid x_{t-n+1:t-1})\) by counting n-gram frequencies in a corpus. Shannon (Shannon, 1948) framed text as a stochastic process and even estimated the entropy of English this way. The fatal weakness is sparsity: most n-grams are never observed, so the bulk of engineering went into smoothing (backoff, Kneser-Ney) to assign probability to unseen sequences. n-gram LMs powered speech recognition and translation for decades, but cannot generalise across similar words — “black car” and “dark car” share no statistics.

Neural probabilistic LM (Bengio 2003). Bengio et al. (Bengio et al., 2003) replaced counts with a feedforward network over distributed word representations (learned embeddings). Similar words get similar vectors, so the model generalises to unseen n-grams — the first crack at the sparsity wall, and the conceptual origin of the word embeddings covered in the RNN era.

Recurrent LMs (Mikolov 2010). Fixed context windows give way to recurrence: an RNN (Mikolov et al., 2010), later LSTM, carries a hidden state so the model can in principle condition on the entire history. This is the RNN-era language model — strong until the vanishing gradient and the sequential bottleneck capped how far context could reach.

The Transformer turn (2017–). Self-attention removed recurrence and let context scale in parallel — and once paired with the scaling laws, next-token prediction became a path to general capability. The rest of this chapter follows that line.

23.3 The modern lineage

23.3.1 One architecture, three scaling steps

The dominant line is a single idea — a causal decoder-only Transformer trained on next-token prediction — scaled in three landmark steps:

  • GPT-2 (Radford et al., 2019) — showed unsupervised next-token pre-training alone yields surprisingly general capability; established byte-level BPE and weight tying as defaults.
  • GPT-3 (Brown et al., 2020) — scaled to 175B parameters and discovered in-context learning: tasks specified by examples in the prompt, no fine-tuning. The moment scale became the strategy.
  • GPT-4 (OpenAI, 2023) — multimodal input, far stronger reasoning, and a shift to not disclosing architecture/size. The frontier became a product, not a paper.
GPT-2 '19 GPT-3 '20 GPT-4 '23 scale + in-context learning PaLM — scale LLaMA — open, Chinchilla Mistral / Mixtral — MoE DeepSeek / Qwen Gemini — multimodal One decoder-only spine; branches mark what each family added — open weights, compute-optimal data, MoE, multimodality.

23.3.2 The branches: what each family added

The lineage is not a single line; each major family contributed a distinct idea (each detailed in its own chapter, catalogued in the landscape):

  • PaLM (Chowdhery et al., 2023) — pushed dense scaling to 540B and mapped emergent capabilities across scale.
  • LLaMA (Touvron et al., 2023) — applied the Chinchilla recipe (smaller model, far more tokens) and opened the weights, igniting the open-LLM ecosystem. Brought RoPE, RMSNorm, SwiGLU into the standard recipe.
  • Mistral / Mixtral (Jiang et al., 2023; Jiang et al., 2024) — strong small dense models, then open Mixture-of-Experts at competitive quality.
  • DeepSeek (DeepSeek-AI, 2024) / Qwen (Bai et al., 2023) — fine-grained MoE, long context, and strong multilingual/coding performance from outside the original Western labs.
  • Gemini (Gemini Team, Google, 2023) — natively multimodal from pre-training, not vision grafted on afterward.

23.3.3 The convergent recipe

Despite different labs, modern LLMs converged on a near-identical stack — itself the strongest evidence that the era’s chapters are the right decomposition:

decoder-only Transformer · RoPE · RMSNorm (pre-norm) · SwiGLU FFN · Chinchilla-scaled data · (increasingly) MoE · SFT + preference alignment · nucleus decoding.

WarningPitfall: open vs closed splits the evidence

Open-weight models (LLaMA, Mistral, DeepSeek, Qwen) publish architecture and often data; frontier closed models (GPT-4, Gemini, Claude) disclose little. Much of what’s “known” about the largest models is inference, not fact. Treat parameter counts and training details for closed models as estimates — and note that the most-cited methods increasingly come from the open ecosystem, which is where this book can point to verifiable detail.

TipGoing deeper

23.4 Application & impact

Family Defining contribution Covered in
GPT-2/3/4 scale → in-context learning → multimodal product BERT & GPT, Scaling Laws
PaLM dense scaling, emergence mapping Scaling Laws
LLaMA open weights, Chinchilla recipe Scaling Laws
Mistral / Mixtral small-dense + open MoE Mixture of Experts
DeepSeek / Qwen fine-grained MoE, long context Mixture of Experts
Gemini native multimodality Multimodal
  • Decoder-only won. The encoder-decoder and encoder-only branches persist for translation and embeddings, but the general-purpose assistant is decoder-only — the GPT bet.
  • The open ecosystem became the research substrate. Most reproducible progress now flows through open-weight families.
  • The recipe stabilised. Convergence on a shared stack means innovation has shifted from architecture to data, scale, alignment, and efficiency.
NoteKey takeaway

The LLM era is one architecture — the decoder-only Transformer — scaled and refined along a few branches: bigger (GPT/PaLM), open and compute-optimal (LLaMA), sparse (Mixtral/DeepSeek), multimodal (Gemini). Each branch is an ingredient from this era’s chapters, assembled — and the chapters that follow (MoE, sub-quadratic attention, multimodality) detail the newest of those branches. This chapter is the lineage; the landscape is the index.

→ Next: Mixture of Experts