47 Deep Reinforcement Learning: DQN to PPO

TL;DR

Deep RL learns a control policy from reward rather than labels: DQN (2013, Q-learning with a deep network) → AlphaGo/AlphaZero (search plus self-play) → policy gradients and PPO (Schulman 2017), the optimizer underneath RLHF.

Depends on: Optimization — GD, SGD, Momentum

47.1 Why this matters

Every model so far in this book learns from a fixed dataset with a known correct answer per example: a label, the next token, a reconstruction target. Reinforcement learning (RL) removes that assumption.

An agent takes actions in an environment and receives a reward signal that may be sparse, delayed, and never tells it directly what the “correct” action was, only how well things turned out. Alignment already uses RL (RLHF) to fine-tune language models against a learned reward signal instead of a fixed label set, and Embodied AI trains policies that act in the physical world; both inherit their optimization machinery from the deep RL lineage in this note.

47.2 The mechanism

47.2.1 Value-based learning: DQN

Q-learning estimates the value of taking action \(a\) in state \(s\): the expected future reward, written \(Q(s,a)\). DQN (Mnih et al., 2013) approximates \(Q(s,a)\) with a deep network trained to minimize the Bellman error, the gap between the current estimate and a one-step-ahead target:

\[ \mathcal{L} = \mathbb{E}\Big[\big(\underbrace{r + \gamma \max_{a'} Q(s', a')}_{\text{target}} - Q(s,a)\big)^2\Big], \]

where \(r\) is the immediate reward, \(\gamma \in (0,1)\) discounts future reward, and \(s'\) is the state reached after taking action \(a\). Once \(Q\) is learned, the policy is simply “take the action with the highest \(Q(s,a)\).” DQN’s headline result was learning to play Atari games directly from raw pixels, with the same reward signal a human player sees (the game score), no hand-engineered features.

47.2.2 Search plus self-play: AlphaGo and AlphaZero

AlphaGo (Silver et al., 2016) combines two learned networks, a value network (how good is this board position) and a policy network (which moves look promising), with Monte Carlo tree search, using the networks to guide which branches of the game tree are worth exploring rather than searching exhaustively. Its successor AlphaZero removed all human game data, learning purely from self-play against earlier versions of itself, and matched or exceeded AlphaGo using the same architecture applied to chess and shogi as well as Go. Search plus self-play, rather than a fixed offline dataset, is what let AlphaZero generalize across games with the same recipe.

47.2.3 Policy gradients and PPO

Rather than learning a value function and deriving a policy from it, policy gradient methods directly parameterize and optimize the policy \(\pi_\theta(a|s)\) to maximize expected reward:

\[ \nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\, A(s,a)\big], \]

where \(A(s,a)\) is the advantage: how much better this action was than the policy’s average action in that state. Plain policy gradients can take a single bad step and destroy a good policy; PPO (Schulman et al., 2017) guards against this by clipping how far one gradient step is allowed to move the policy away from its previous version. This training stability, more than raw sample efficiency, is why PPO became the default choice for RLHF: fine-tuning a large pretrained language model against a reward model tolerates very little instability before the model’s fluency degrades.

import numpy as np
rng = np.random.default_rng(0)

def clipped_objective(ratio, advantage, eps=0.2):
    unclipped = ratio * advantage
    clipped = np.clip(ratio, 1 - eps, 1 + eps) * advantage
    return np.minimum(unclipped, clipped)   # PPO's pessimistic bound

ratios = np.array([0.7, 1.0, 1.5, 2.0])   # new_policy_prob / old_policy_prob
advantage = 1.0                            # this action was better than average
print("PPO objective:", np.round(clipped_objective(ratios, advantage), 3))

PPO objective: [0.7 1.  1.2 1.2]

47.2.4 What to observe

DQN learns values and derives a policy from them; PPO learns the policy directly. Both are valid ways to convert a reward signal into behavior.
AlphaZero’s self-play removes the need for any human-labeled dataset entirely, generating its own curriculum by playing against itself.
PPO’s clipping is a stability mechanism, not an accuracy improvement: it trades some update speed for guaranteed-bounded policy changes, exactly the property RLHF fine-tuning needs.

47.3 Application & impact

Concept here	What it became	Where you see it today
DQN’s value-based control	Deep RL from raw sensory input	Robotics and game-playing agents
AlphaZero’s self-play	Learning without human demonstration data	A recurring idea in agentic self-improvement research
Policy gradients	Direct policy optimization	The basis for RLHF
PPO’s clipped update	Stable fine-tuning against a reward model	Alignment & Instruction Tuning’s RLHF stage

Key takeaway

Deep RL supplies the optimization machinery for learning from reward instead of labels, value-based (DQN), search-based (AlphaZero), or policy-based (PPO). PPO’s contribution, a stable, bounded policy update, is precisely why it, rather than an older or more sample-efficient method, became the standard tool for RLHF and for embodied policy fine-tuning.

← Back to: Alignment & Instruction Tuning · Embodied AI & VLA