13  Convolutional Seq2Seq: CNNs for Sequence Modeling

TipTL;DR

Before the Transformer, Gehring et al. (2017) showed that stacked 1D convolutions — not RNNs — could match or beat the best recurrent Seq2Seq models on translation, at a fraction of the training time. The key: convolutions process all positions in parallel; the sequential bottleneck of the RNN is gone. ConvSeq2Seq beat RNN Seq2Seq on WMT14 en→de. Three months later, Vaswani et al. published the Transformer.

Depends on: Seq2Seq RNN Encoder-Decoder

13.1 Why this matters

RNNs process sequences left to right — step \(t\) cannot begin until step \(t-1\) is complete. On modern GPU hardware, which is built for massive parallelism, this is a structural bottleneck: a 100-token sequence requires 100 sequential LSTM steps, regardless of batch size or hardware. ConvSeq2Seq proves the bottleneck is not fundamental to sequence modeling — it is specific to the recurrent architecture.

The lesson that carries forward into the Transformer: parallelism enables scale. If every position can be computed simultaneously, training time falls from \(O(T)\) to \(O(1)\) in depth, enabling larger models, larger batches, and longer sequences.

13.2 The mechanism

13.2.1 1D convolution over a sequence

A 1D convolution with kernel width \(k\) and \(d\) output channels maps a sequence of vectors \((x_1, \ldots, x_T) \in \mathbb{R}^{T \times d}\) to an output sequence of the same length by sliding a weight matrix \(\mathbf{W} \in \mathbb{R}^{kd \times 2d}\) over every window of \(k\) consecutive positions:

\[ z_t = \mathbf{W} [x_{t-k/2}; \ldots; x_{t+k/2}] + \mathbf{b} \in \mathbb{R}^{2d}. \]

Padding keeps the sequence length constant across layers.

13.2.2 Gated Linear Unit (GLU)

Gehring et al. (Gehring et al., 2017) apply a Gated Linear Unit to the convolution output — split \(z_t\) into two halves and gate one with the other:

\[ \text{GLU}(z_t) = z_t^{(1)} \odot \sigma(z_t^{(2)}) \in \mathbb{R}^d. \]

The gate \(\sigma(z_t^{(2)})\) controls how much of the linear output \(z_t^{(1)}\) passes through, analogous to the LSTM’s output gate but applied position-wise.

13.2.3 Receptive field grows with depth

A single conv layer with kernel \(k\) sees \(k\) consecutive positions. Stack \(N\) layers: the receptive field grows to \(k + (k-1)(N-1)\). With \(k = 5\) and \(N = 20\), every output position sees 81 input positions — enough for most practical sentence lengths.

token positions (all processed in parallel) x₁ x₂ x₃ x₄ x₅ x₆ x₇ x₈ Conv layer 1 (k=3, receptive field = 3) Conv layer 2 (k=3, receptive field = 5) Conv layer 3 (k=3, receptive field = 7)

13.2.4 Multi-hop attention

Between each decoder convolution layer and the encoder output, ConvSeq2Seq adds an attention step — a weighted sum over encoder states, recomputed per layer. With \(N\) decoder layers, the decoder performs \(N\) attention lookups per output token. This multi-hop attention approximates the global context an LSTM accumulates over the full source.

13.2.5 Minimal sketch

A single 1D conv + GLU step in NumPy, applied to all positions at once.

import numpy as np

rng = np.random.default_rng(2)
T, d, k = 8, 16, 3    # seq length, channels, kernel width

# Weight matrix: maps k*d inputs → 2d outputs (for GLU)
W = rng.normal(scale=0.1, size=(2 * d, k * d))
b = np.zeros(2 * d)

def conv_glu(X):
    """X: (T, d) → (T, d) via 1D conv + GLU, same-padding."""
    pad = k // 2
    X_pad = np.pad(X, ((pad, pad), (0, 0)))
    out = np.zeros((T, d))
    for t in range(T):
        window = X_pad[t:t + k].ravel()   # k*d
        z = W @ window + b                 # 2d
        out[t] = z[:d] * (1 / (1 + np.exp(-z[d:])))   # GLU
    return out

X = rng.normal(size=(T, d))
Y = conv_glu(X)
print(f"input shape: {X.shape} → output shape: {Y.shape}")
print(f"all positions computed simultaneously: T={T} steps → O(1) parallel depth")
input shape: (8, 16) → output shape: (8, 16)
all positions computed simultaneously: T=8 steps → O(1) parallel depth
WarningPitfall: fixed receptive field

A stack of \(N\) conv layers with kernel \(k\) has a maximum receptive field of \(k + (k-1)(N-1)\). Sentences longer than this window are invisible to the model across the full depth. Self-attention — which attends to all positions from every layer — removes this constraint entirely, which is why the Transformer superseded ConvSeq2Seq within months.

13.3 Application & impact

Concept here What it became Where
Parallel sequence processing Multi-head self-attention Transformer (2017)
GLU activation SwiGLU, GeGLU LLaMA, PaLM, Gemini
Multi-hop attention Multi-layer cross-attention Transformer decoder
Residual connections Unchanged Every deep network since ResNet
  • ConvSeq2Seq (May 2017) set a new WMT14 en→de BLEU record.
  • The Transformer (June 2017) surpassed it within weeks and introduced unlimited receptive field via self-attention, ending the conv-vs-RNN debate.
  • GLU activations are a direct legacy: SwiGLU (used in LLaMA, PaLM, and Gemini) is a gated linear unit with a Swish gate instead of sigmoid.
NoteKey takeaway

ConvSeq2Seq proved that the sequential constraint of RNNs was not necessary for sequence modeling — full parallelism is achievable. It did not solve the fixed receptive-field limit. The Transformer solved both by replacing the convolution window with attention over all positions simultaneously.

→ Next: Additive and Multiplicative Attention