10  Vanilla RNN & BPTT

TipTL;DR

An RNN processes a sequence step by step, sharing the same weights at every position. The hidden state \(h_t = \tanh(\mathbf{W}_h h_{t-1} + \mathbf{W}_x x_t + \mathbf{b})\) is a running summary of everything seen so far. Training by BPTT unrolls the network across \(T\) steps and runs standard backprop — but the gradient reaching early steps is a product of \(T\) Jacobians, which collapses toward zero. That is not a bug in the algorithm; it is a structural limit of the architecture.

Depends on: Word Embeddings

10.1 Why this matters

An MLP requires a fixed-size input. Language, speech, and DNA are variable-length sequences where order matters — the word at position 3 means something different depending on positions 1 and 2. An RNN handles this with two ideas:

  • Weight sharing across time. The same \(\mathbf{W}_h\), \(\mathbf{W}_x\) apply at every step, so the model has a fixed parameter count regardless of sequence length.
  • Recurrent hidden state. \(h_t\) carries context forward; each step conditions on the entire history compressed into \(h_{t-1}\).

Rumelhart, Hinton & Williams (Rumelhart et al., 1986) introduced BPTT as a direct consequence of backprop on unrolled networks. Elman (Elman, 1990) popularised the Simple RNN (SRN) as a cognitive model of sequential structure. Werbos (Werbos, 1990) formalized BPTT independently.

10.2 The mechanism

10.2.1 RNN cell

At each step \(t\), the cell maps \((h_{t-1}, x_t)\) to a new hidden state:

\[ h_t = \tanh\!\bigl(\mathbf{W}_h h_{t-1} + \mathbf{W}_x x_t + \mathbf{b}\bigr), \qquad \hat{y}_t = \mathbf{W}_y h_t + \mathbf{b}_y. \]

The same three weight matrices \((\mathbf{W}_h, \mathbf{W}_x, \mathbf{W}_y)\) are reused at every step — unrolling creates a deep network with tied weights.

10.2.2 Unrolled computation graph

RNN RNN RNN h₀ h₁ h₂ h_T x₁ x₂ x_T ŷ BPTT — gradient flows backward through time

10.2.3 BPTT: backprop on the unrolled graph

The total loss \(\mathcal{L} = \sum_t \mathcal{L}_t\) has gradients:

\[ \frac{\partial \mathcal{L}}{\partial \mathbf{W}_h} = \sum_{t=1}^T \sum_{k=1}^t \frac{\partial \mathcal{L}_t}{\partial h_t} \underbrace{\left(\prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}}\right)}_{\text{Jacobian product}} \frac{\partial h_k}{\partial \mathbf{W}_h}. \]

Each Jacobian is:

\[ \frac{\partial h_i}{\partial h_{i-1}} = \mathbf{W}_h^\top \operatorname{diag}\!\bigl(1 - h_i^2\bigr). \]

For \(t - k \gg 1\), this product of matrices either vanishes (if the singular values of \(\mathbf{W}_h\) are \(< 1\)) or explodes (if \(> 1\)). In practice, tanh saturates and the gradient signal from early positions reaches \(\mathbf{W}_h\) as near-zero — the model cannot learn long-range dependencies.

WarningPitfall: gradient clipping is not a fix

Clipping the gradient norm before each update (the standard mitigation for explosion) stabilises training but does not cure the vanishing direction. The network still cannot credit or blame weights for events many steps back. The architectural fix is the LSTM — covered in the next chapter.

10.2.4 Minimal sketch

The sketch below runs a single RNN cell forward across a short sequence, then computes gradients manually to make the Jacobian product visible.

import numpy as np

rng = np.random.default_rng(0)
T, d_x, d_h = 5, 4, 8

Wh = rng.normal(scale=0.1, size=(d_h, d_h))
Wx = rng.normal(scale=0.1, size=(d_h, d_x))
b  = np.zeros(d_h)

def rnn_step(h, x):
    return np.tanh(Wh @ h + Wx @ x + b)

# Forward pass — track hidden states
X = rng.normal(size=(T, d_x))
h = np.zeros(d_h)
hs = [h]
for t in range(T):
    h = rnn_step(h, X[t])
    hs.append(h)

# Jacobian norm at each step (how much gradient survives going back one step)
for t in range(1, T + 1):
    J = Wh.T * (1 - hs[t] ** 2)      # diag(1-h²) × Wh
    print(f"step {t}  ‖∂h_t/∂h_{{t-1}}‖ = {np.linalg.norm(J):.4f}")
step 1  ‖∂h_t/∂h_{t-1}‖ = 0.6897
step 2  ‖∂h_t/∂h_{t-1}‖ = 0.7095
step 3  ‖∂h_t/∂h_{t-1}‖ = 0.6944
step 4  ‖∂h_t/∂h_{t-1}‖ = 0.6795
step 5  ‖∂h_t/∂h_{t-1}‖ = 0.7193
TipGoing deeper

10.3 Application & impact

Concept here What it becomes Where
Weight sharing across time Shared kernel in 1D convolution ConvSeq2Seq (2017)
Hidden state as memory Cell state + gates LSTM (1997), GRU (2014)
BPTT Truncated BPTT, gradient clipping Every RNN training loop
Sequential bottleneck The problem attention solves Bahdanau 2015, Transformer 2017
  • The vanilla RNN was the dominant sequence model from 1990 to ~1997, when LSTMs superseded it for tasks requiring long-range memory.
  • Truncated BPTT — stopping the backward pass after \(k\) steps — is the standard practical variant; it limits memory and avoids the worst of vanishing, at the cost of ignoring very long dependencies.
  • Every RNN training loop today (PyTorch nn.RNN) is BPTT under the hood.
NoteKey takeaway

The RNN’s weight sharing solves variable-length inputs; BPTT makes it trainable. The Jacobian product that BPTT must multiply is the same product that causes vanishing gradients — both are intrinsic to the architecture, not implementation bugs. The next chapter shows how the LSTM breaks this constraint by architectural design.

→ Next: LSTM & GRU