Depends on: Optimization — GD, SGD, Momentum
A network with enough parameters can memorize its training set perfectly and still fail on new data. That gap between training loss and test loss is overfitting. The four techniques here fight it by four different mechanisms: inject noise into training (dropout), penalize large weights directly (weight decay), enlarge the effective training set (augmentation), or simply stop before the model has memorized too much (early stopping).
Dropout (Srivastava et al., 2014) randomly zeroes each hidden unit with probability \(p\) during training, then rescales the survivors so the expected activation magnitude stays unchanged:
\[ \tilde{h}_i = \frac{m_i}{1-p}\, h_i, \qquad m_i \sim \text{Bernoulli}(1-p). \]
The \(\frac{1}{1-p}\) rescaling is why dropout needs no adjustment at inference: it is simply turned off, and the full network is used as-is. The effect on training is that no single unit can rely on any other specific unit being present, so every unit learns features useful in combination with many different random subsets of the rest of the network. This is what gives dropout its informal description as training an implicit ensemble of subnetworks that share weights.
Weight decay adds a penalty on the squared magnitude of the weights to the loss:
\[ \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{data}} + \lambda \lVert \boldsymbol{\theta} \rVert_2^2, \qquad \lambda \text{ controls how strongly large weights are punished.} \]
Smaller weights correspond to a simpler function, one less sensitive to any single input feature, so penalizing weight magnitude biases the optimizer toward simpler solutions. This is the same \(\lambda\theta\) term AdamW decouples from the adaptive gradient scaling; the idea of weight decay predates that correction and is simpler than it.
Two lighter-weight regularizers round out the toolbox.
import numpy as np
rng = np.random.default_rng(0)
def dropout(h, p, training=True):
if not training:
return h
mask = rng.binomial(1, 1 - p, size=h.shape)
return mask * h / (1 - p)
h = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
print("train (p=0.5):", np.round(dropout(h, 0.5), 2))
print("eval: ", dropout(h, 0.5, training=False))train (p=0.5): [ 2. 0. 0. 0. 10.]
eval: [1. 2. 3. 4. 5.]| Concept here | What it became | Where you see it today |
|---|---|---|
| Dropout | Standard regularizer for dense/RNN layers | Less common in modern Transformers at scale (data volume regularizes instead) |
| Weight decay | The \(\lambda\theta\) term in every modern optimizer | AdamW, used to train nearly every LLM |
| Data augmentation | Vision and NLP training-set expansion | Standard in vision pipelines; back-translation in NLP |
| Early stopping | Universal, free regularizer | Every training loop with a validation split |
← Back to: MLP & Backpropagation