22  Alignment & Instruction Tuning

TipTL;DR

A scaled base model predicts likely text — not helpful, honest, or safe answers. Alignment closes that gap in stages: supervised fine-tuning (SFT) teaches the instruction-following format; RLHF or DPO optimises toward human preferences. DPO’s insight is that you can skip the reward model and reinforcement learning entirely and fit preferences with a simple classification-style loss. Separately, retrieval-augmented generation (RAG) attacks a different failure — factual grounding — by retrieving documents at inference instead of trusting parametric memory.

Depends on: Scaling Laws & Emergence

22.1 Why this matters

Scaling gives you a model that completes text. Ask GPT-3 a question and it might continue with more questions — because that’s a likely continuation, not because it’s unhelpful on purpose. The pre-training objective (predict the next token) is simply not the objective we want (be helpful, truthful, harmless).

Alignment is the bridge from “language model” to “assistant.” It is what turned GPT-3 into ChatGPT — a far smaller change in scale than in behaviour — and it is now as central to building a usable model as pre-training itself.

22.2 The mechanism

22.2.1 Stage 1: Supervised fine-tuning (SFT)

Collect a dataset of (instruction, ideal response) pairs written by humans and fine-tune the base model on them with ordinary next-token loss. This teaches the format of instruction-following: respond to the request, don’t continue it. SFT alone produces a usable instruction model — but it can only imitate demonstrations, and good demonstrations are expensive and inconsistent.

22.2.2 Stage 2: Learning from preferences (RLHF)

Humans find it far easier to compare two responses than to write the ideal one. RLHF (Ouyang et al., 2022) exploits this:

  1. Reward model. Collect human rankings of response pairs; train a model \(r_\phi(x, y)\) to score responses so preferred ones score higher.
  2. RL optimisation. Fine-tune the policy with PPO (Schulman et al., 2017) to maximise reward, with a KL penalty keeping it close to the SFT model (so it doesn’t collapse into reward-hacking gibberish): \[\max_\pi\; \mathbb{E}\,[\,r_\phi(x,y)\,] - \beta\,\mathrm{KL}\!\big(\pi \,\|\, \pi_{\text{SFT}}\big).\]

This is the recipe behind InstructGPT and ChatGPT. It works, but the pipeline is fragile: a separate reward model, an RL loop, and reward-hacking to fight.

22.2.3 Stage 3: DPO: preferences without RL

Direct Preference Optimization (Rafailov et al., 2023) shows the RLHF objective has a closed-form optimal policy, which lets you skip the reward model and PPO entirely. You optimise the policy directly on preference pairs \((y_w \succ y_l)\) with a simple logistic loss:

\[ \mathcal{L}_{\text{DPO}} = -\,\mathbb{E}\left[ \log \sigma\!\left( \beta \log \tfrac{\pi(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \tfrac{\pi(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right)\right]. \]

Same goal — raise the probability of preferred responses relative to a frozen reference — but it’s just supervised learning. DPO’s stability and simplicity made it the default open-source alignment method.

Base model predicts text SFT follows format RLHF (reward+PPO) —or— DPO optimise human preferences From completion engine to assistant DPO collapses the reward-model + RL stage into one supervised preference loss.

22.2.4 Retrieval-Augmented Generation (RAG)

Alignment fixes behaviour; it does nothing for facts. A model’s knowledge is frozen at training time and stored lossily in its weights (parametric memory), so it goes stale and hallucinates confident wrong answers. RAG (Lewis et al., 2020) adds non-parametric memory: an external corpus the model reads at inference.

The pipeline is retrieve-then-generate:

  1. Retrieve. Embed the query, search a vector index of document chunks for the top-\(k\) most similar passages.
  2. Augment. Concatenate the retrieved passages into the prompt as context.
  3. Generate. The LLM answers grounded in the supplied passages, ideally citing them.

Why it’s a paradigm shift, not a patch:

  • Updatable knowledge — swap the index, not the weights; no retraining to add today’s facts.
  • Grounding & attribution — answers trace to sources, which both reduces hallucination and makes it auditable.
  • Smaller models, current facts — knowledge lives in the corpus, so the parameters can be spent on reasoning rather than memorisation.
WarningPitfall: RAG fails at retrieval, not generation

A RAG system is only as good as what it retrieves. If the retriever misses the relevant passage, the model confidently answers from parametric memory anyway — you’ve added latency without grounding. Most RAG failures are retrieval failures (bad chunking, weak embeddings, no reranking), not generation failures. Grounding is a property of the whole pipeline, not the LLM.

TipGoing deeper
  • Scaling parameters without scaling per-token compute — Mixture of Experts.
  • The full landscape of alignment and retrieval variants (Constitutional AI, RLAIF, agentic retrieval) — see the Research Landscape.

22.2.5 Minimal sketch

The DPO loss on a single preference pair, in NumPy.

import numpy as np

def dpo_loss(logp_w, logp_l, ref_w, ref_l, beta=0.1):
    # logp_*: policy log-probs; ref_*: frozen reference log-probs
    margin = beta * ((logp_w - ref_w) - (logp_l - ref_l))
    return -np.log(1 / (1 + np.exp(-margin)))   # -log sigmoid(margin)

# preferred response gains probability over reference; rejected loses it
print("aligned pair  :", round(dpo_loss(-2.0, -5.0, -3.0, -3.0), 3))   # low loss
print("misaligned    :", round(dpo_loss(-5.0, -2.0, -3.0, -3.0), 3))   # high loss
aligned pair  : 0.554
misaligned    : 0.854

22.3 Application & impact

Method What it adds Trade-off
SFT instruction-following format limited to demonstration quality
RLHF optimises human preference fragile pipeline, reward hacking
DPO preferences without RL simpler, now the open-source default
RAG factual grounding, fresh knowledge only as good as retrieval
  • RLHF made ChatGPT possible — the leap from GPT-3 to a usable assistant was mostly alignment, not scale.
  • DPO democratised alignment — a stable supervised loss any lab can run put preference tuning within reach of open models.
  • RAG decoupled knowledge from weights — the dominant pattern for enterprise LLMs, where facts must be current, sourced, and private.
NoteKey takeaway

A base model is a completion engine; alignment makes it an assistant. SFT teaches the format, RLHF/DPO optimise toward human preference (DPO by collapsing the reward model and RL into one supervised loss). RAG is the orthogonal fix: ground the model in retrieved sources so it stays current and stops hallucinating. Together they turn a predictor of likely text into a helpful, grounded system.

→ Next: Language Models — The Lineage