32  Modern Activations: ReLU, GELU, SiLU, SwiGLU

← Back to: Activations & Non-linearity

TipTL;DR

The foundations note showed why sigmoid and tanh choke gradients in deep stacks. The fix was to stop saturating: ReLU \(\max(0,x)\) passes the positive half unchanged, and its descendants — GELU, SiLU/Swish, and the gated SwiGLU — smooth the kink and become the default non-linearity inside every Transformer FFN.

32.1 From saturation to rectification

Sigmoid/tanh saturate on both tails, so their derivative vanishes and deep gradients shrink geometrically. ReLU (Nair & Hinton, 2010) removes the problem on the positive side outright:

\[ \mathrm{ReLU}(x) = \max(0, x), \qquad \mathrm{ReLU}'(x) = \begin{cases} 1 & x > 0 \\ 0 & x < 0 \end{cases}. \]

For \(x>0\) the gradient is exactly \(1\) — no shrinkage, no matter the depth. This single property, paired with good initialization, is what made very deep networks (AlexNet onward) trainable. The cost is the dead-ReLU failure mode: a unit stuck at \(x<0\) for all inputs receives zero gradient forever.

32.2 The smooth descendants

ReLU’s hard zero at the origin is non-differentiable and discards all negative information. Three smooth variants soften it, and these are what Transformers actually use:

\[ \mathrm{GELU}(x) = x\,\Phi(x), \qquad \mathrm{SiLU}(x) = x\,\sigma(x), \]

where \(\Phi\) is the standard-normal CDF and \(\sigma\) the logistic sigmoid. Both gate the input by a smooth, input-dependent factor — small negatives leak through (non-zero gradient), large positives approach identity.

  • GELU (Hendrycks & Gimpel, 2016) — weights the input by its probability under a Gaussian; the default in BERT, GPT-2/3, and most encoder/decoder stacks.
  • SiLU / Swish (Ramachandran et al., 2017) — \(x\,\sigma(x)\), found by activation search; nearly identical in shape to GELU and cheaper to compute.
x ReLU GELU / SiLU (smooth, small leak)

32.3 SwiGLU: the gated FFN variant

Modern LLM feed-forward blocks rarely apply a bare activation. They use a gated linear unit: split the projection into two halves, pass one through an activation, and multiply (Shazeer, 2020):

\[ \mathrm{SwiGLU}(x) = \big(\mathrm{SiLU}(xW_1)\big) \odot (xW_2), \]

then project back with \(W_3\). The multiplicative gate gives the FFN a data-dependent on/off control per channel — empirically a consistent quality gain, at the cost of a third weight matrix (hidden width is scaled down to keep parameters matched). This is the FFN used in LLaMA, Mistral, and PaLM — the “SwiGLU FFN” named in the language-model convergent recipe.

import numpy as np
def relu(x): return np.maximum(0, x)
def gelu(x): return 0.5 * x * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715*x**3)))
def silu(x): return x / (1 + np.exp(-x))

x = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])
print("ReLU:", relu(x))
print("GELU:", np.round(gelu(x), 3))
print("SiLU:", np.round(silu(x), 3))   # note the small negative leak at x=-1
ReLU: [0. 0. 0. 1. 3.]
GELU: [-0.004 -0.159  0.     0.841  2.996]
SiLU: [-0.142 -0.269  0.     0.731  2.858]
NoteKey takeaway

The activation story is one of not saturating: ReLU fixed the gradient by passing positives unchanged; GELU/SiLU smoothed its kink and let a little negative signal through; SwiGLU added a multiplicative gate. The foundational sigmoid/tanh survive only as gates (LSTM, attention) — the hidden-layer non-linearity of the deep-net and Transformer eras is this rectified lineage.

← Back to: Activations & Non-linearity · Used in: The Transformer Architecture (FFN)