Depends on: Information Theory
A neuron is a weighted sum followed by a non-linearity. The artificial-neuron chapter focused on the weighted sum; this one is about why the non-linearity is not optional. Strip it out and a deep network is algebraically identical to a shallow one — no amount of stacking buys representational power. The non-linearity is the single ingredient that lets composition build complexity.
It also matters because the choice of activation shapes how gradients flow. The early activations (sigmoid, tanh) saturate, and that saturation is exactly what makes deep networks hard to train — the bridge to gradient flow.
A linear layer is \(\mathbf{h} = \mathbf{W}\mathbf{x}\). Stack two:
\[ \mathbf{y} = \mathbf{W}_2(\mathbf{W}_1\mathbf{x}) = (\mathbf{W}_2\mathbf{W}_1)\mathbf{x} = \mathbf{W}\mathbf{x}. \]
The composition is itself a single linear map \(\mathbf{W} = \mathbf{W}_2\mathbf{W}_1\). No depth, no gain. Inserting a non-linearity \(\phi\) between layers breaks this collapse — the network can now represent functions no single linear map can.
import numpy as np
rng = np.random.default_rng(0)
W1 = rng.normal(size=(4, 2))
W2 = rng.normal(size=(1, 4))
x = rng.normal(size=(2,))
two_layers = W2 @ (W1 @ x) # stacked, no non-linearity
one_layer = (W2 @ W1) @ x # a single equivalent linear map
print("two linear layers:", two_layers)
print("single equivalent:", one_layer)
print("identical:", np.allclose(two_layers, one_layer))two linear layers: [1.23783565]
single equivalent: [1.23783565]
identical: TrueThe two foundational activations:
\[ \sigma(z) = \frac{1}{1 + e^{-z}} \in (0, 1), \qquad \tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}} \in (-1, 1). \]
Their derivatives have clean closed forms:
\[ \sigma'(z) = \sigma(z)\,(1 - \sigma(z)) \le \tfrac{1}{4}, \qquad \tanh'(z) = 1 - \tanh^2(z) \le 1. \]
Two things to note. First, both saturate: for large \(|z|\) the output flattens and the derivative goes to zero. Second, tanh is zero-centered while sigmoid is not — sigmoid’s all-positive outputs bias the gradients of the next layer, which is why tanh is generally preferred as a hidden activation (LeCun et al., 1998).
import matplotlib.pyplot as plt
z = np.linspace(-6, 6, 300)
sig = 1 / (1 + np.exp(-z)); th = np.tanh(z)
dsig = sig * (1 - sig); dth = 1 - th**2
fig, (a, b) = plt.subplots(1, 2, figsize=(8, 3))
a.plot(z, sig, label="sigmoid"); a.plot(z, th, "--", label="tanh")
a.set_title("activation"); a.legend(); a.grid(alpha=0.3)
b.plot(z, dsig, label="sigmoid'"); b.plot(z, dth, "--", label="tanh'")
b.set_title("derivative"); b.legend(); b.grid(alpha=0.3)
plt.tight_layout(); plt.show()
Sigmoid and tanh are no longer the default hidden activation — modern nets use ReLU and its descendants (deferred to their timeline slots). But neither is legacy: both survive in specific, load-bearing roles.
| Concept here | What it became | Where you see it today |
|---|---|---|
| sigmoid | gating function (squash to \((0,1)\)) | LSTM input/forget/output gates; GLU/attention gates |
| tanh | bounded state activation | LSTM cell-state update |
| sigmoid output | probability head | binary classification output (with BCE from ch 02) |
| saturation problem | motivation for non-saturating activations | ReLU (2010), GELU (2016) — deferred |
| non-linearity requirement | universal approximation | every deep network; formalized with the MLP next chapter |
→ Next: MLP & Backpropagation