12 Seq2Seq RNN Encoder-Decoder

TL;DR

Two LSTMs end-to-end: an encoder reads the entire source sequence and compresses it to a fixed vector \(c = h_T^{\text{enc}}\); a decoder generates the target token by token, conditioned on \(c\). Sutskever, Vinyals & Le (2014) used this to build the first neural MT system competitive with phrase-based pipelines. The fixed context vector is both the architecture’s breakthrough and its bottleneck — the problem attention was built to solve.

Depends on: LSTM & GRU

12.1 Why this matters

Before Seq2Seq, natural language processing was a pipeline: tokenise, parse, align, translate, reorder, score. Each component was separately engineered and separately trained. A single differentiable model that maps an entire input sequence to an entire output sequence — trained end-to-end on (source, target) pairs — was architecturally new.

Three things Seq2Seq introduced that every downstream model inherits:

Encoder-decoder abstraction. The encoder produces a representation; the decoder consumes it. Attention, the Transformer, and T5 all follow this split.
Autoregressive decoding. The decoder generates one token at a time, conditioning each token on all previous outputs.
End-to-end differentiability. The entire system — encoder, decoder, output projection — is trained jointly on cross-entropy.

12.2 The mechanism

12.2.1 Encoder

The encoder is a stack of LSTMs that reads the source sequence left to right:

\[ h_t^{\text{enc}}, c_t^{\text{enc}} = \text{LSTM}\!\bigl(h_{t-1}^{\text{enc}},\, c_{t-1}^{\text{enc}},\, x_t\bigr). \]

The context vector is the final hidden state: \(\mathbf{c} = h_{T_x}^{\text{enc}}\).

Sutskever et al. (Sutskever et al., 2014) found that reversing the source sequence before encoding improved translation quality — putting the beginning of the source closer to the beginning of the target in the unrolled graph shortens the gradient path.

12.2.2 Decoder

The decoder generates the target sequence autoregressively:

\[ h_t^{\text{dec}}, c_t^{\text{dec}} = \text{LSTM}\!\bigl(h_{t-1}^{\text{dec}},\, c_{t-1}^{\text{dec}},\, [y_{t-1};\, \mathbf{c}]\bigr), \] \[ P(y_t \mid y_{<t}, \mathbf{c}) = \operatorname{softmax}\!\bigl(\mathbf{W}_o h_t^{\text{dec}} + \mathbf{b}_o\bigr). \]

The embedding of the previous target token \(y_{t-1}\) and the context vector \(\mathbf{c}\) are concatenated as the decoder’s input at each step. At inference, \(y_{t-1}\) is the model’s own previous prediction (greedy or beam-search).

12.2.3 Teacher forcing and exposure bias

At training time, the decoder receives the gold target token \(y_{t-1}\) as input at each step, regardless of its own prediction. This makes gradients stable and training fast.

At inference time, there is no gold input — the decoder must feed its own previous prediction \(\hat{y}_{t-1}\). The mismatch between training distribution (perfect prefix) and inference distribution (own predictions, possibly wrong) is called exposure bias. Errors accumulate: a wrong token at step 3 corrupts step 4’s input, compounding throughout the sequence.

Pitfall: the fixed context bottleneck

\(\mathbf{c} = h_{T_x}^{\text{enc}}\) must encode the entire source sentence in a vector of fixed size \(d_h\). For short sentences this is fine; for long sentences, information is lost. Translation quality degrades sharply beyond ~20 tokens. This is the problem Additive and Multiplicative Attention solves by replacing \(\mathbf{c}\) with a dynamic, per-step weighted sum over all encoder hidden states.

12.2.4 Minimal sketch

A minimal NumPy Seq2Seq: encode a source sequence to a context vector, then decode greedily using the context at every step.

import numpy as np

rng = np.random.default_rng(1)
d_x, d_h, V = 8, 16, 20   # embed dim, hidden dim, vocab size

def sigmoid(z): return 1 / (1 + np.exp(-z))

# Shared LSTM step (same structure for encoder and decoder)
W = rng.normal(scale=0.1, size=(4 * d_h, d_h + d_x))
b = np.zeros(4 * d_h)

def lstm_step(h, c, x):
    f, i, g, o = np.split(W @ np.concatenate([h, x]) + b, 4)
    c = sigmoid(f) * c + sigmoid(i) * np.tanh(g)
    return sigmoid(o) * np.tanh(c), c

# Encoder
source = rng.normal(size=(6, d_x))       # 6-token source sequence
h, c = np.zeros(d_h), np.zeros(d_h)
for x in source:
    h, c = lstm_step(h, c, x)
context = h                               # fixed context vector

# Decoder — greedy, feed context at every step
Wo = rng.normal(scale=0.1, size=(V, d_h))
E  = rng.normal(scale=0.01, size=(d_x, V))  # target token embeddings

h, c = context, np.zeros(d_h)
token = 0                                  # <BOS> token id
output = []
for _ in range(4):                         # decode 4 steps
    x = np.concatenate([E[:, token], context])[:d_x]   # embed + context
    h, c = lstm_step(h, c, x)
    token = (Wo @ h).argmax()
    output.append(token)

print("decoded token ids:", output)

decoded token ids: [np.int64(14), np.int64(7), np.int64(14), np.int64(7)]

12.3 Application & impact

Concept here	What it became	Where
Encoder-decoder split	Universal abstraction	Transformer, T5, BART, every MT system
Fixed context vector	Dynamic context (attention)	Bahdanau 2015 → Transformer 2017
Autoregressive decoding	Unchanged	Every LLM (GPT, LLaMA, Gemini)
Teacher forcing	Standard training recipe	All seq-to-seq models
Exposure bias	Scheduled sampling, RL fine-tuning	MIXER, REINFORCE, DPO

Cho et al. (Cho et al., 2014) introduced the encoder-decoder framing for phrase representations in the same year; their GRU-based variant is the direct predecessor of attention-based MT.
Seq2Seq with beam search became the dominant NMT architecture from 2014–2016, before attention made the fixed context bottleneck obsolete.
The encoder-decoder abstraction outlived the RNN: the Transformer adopts it unchanged, replacing recurrence with self-attention.

Key takeaway

Seq2Seq made neural MT viable by collapsing a multi-component pipeline into a single differentiable model. The architecture it introduced — encoder compresses source, decoder generates target — remains the dominant framing for structured generation. Its one structural weakness, the fixed context bottleneck, is precisely the problem the next chapters address.

→ Next: Convolutional Seq2Seq