Depends on: The Artificial Neuron
The perceptron’s update rule reacts to a 0/1 mistake: right or wrong, nothing in between. That signal has no gradient — you cannot ask “how much should I change \(\mathbf{w}\) to be a little less wrong?” Every method after the perceptron trains by gradient descent on a smooth loss, which means a differentiable measure of how wrong a probabilistic prediction is. That measure is cross-entropy, and its foundations were laid by Shannon (Shannon, 1948) in 1948 — chronologically before the perceptron, which is why it sits here, early in the timeline.
The treatment here focuses on binary classification (the perceptron’s own domain). The multi-class generalization (softmax + categorical cross-entropy) is introduced later, where multi-output models first appear.
Shannon’s self-information of an event with probability \(p\) is \(-\log p\): rare events are surprising, certain events (\(p = 1\)) carry zero surprise. Entropy is the expected surprise of a distribution:
\[ H(p) = -\sum_x p(x) \log p(x). \]
For a binary outcome with \(P(\text{1}) = p\), entropy peaks at \(p = 0.5\) (maximum uncertainty) and is zero at \(p \in \{0, 1\}\) (no uncertainty).
If the true distribution is \(p\) but the model predicts \(q\), the expected cost of encoding outcomes with \(q\) is the cross-entropy:
\[ H(p, q) = -\sum_x p(x) \log q(x). \]
It is minimized, for fixed \(p\), exactly when \(q = p\). As a training loss, \(p\) is the (known) label distribution and \(q\) is the model’s prediction.
For a label \(y \in \{0, 1\}\) and a model prediction \(q = P(y = 1)\), the true distribution puts all mass on the observed label, so the sum collapses to a single, clean expression:
\[ \mathcal{L}_{\text{BCE}}(y, q) = -\big[\, y \log q + (1 - y)\log(1 - q)\,\big]. \]
A confident-correct prediction (\(q \to 1\) when \(y = 1\)) drives the loss to \(0\); a confident-wrong one (\(q \to 0\) when \(y = 1\)) drives it to \(+\infty\). That asymmetry — gentle on correct, brutal on confidently wrong — is what makes the gradient informative.
The gap between cross-entropy and the irreducible entropy of \(p\) is the Kullback–Leibler divergence (Kullback & Leibler, 1951):
\[ D_{\mathrm{KL}}(p \,\|\, q) = \sum_x p(x)\log\frac{p(x)}{q(x)} = H(p, q) - H(p) \;\ge\; 0. \]
Since \(H(p)\) is fixed by the data, minimizing cross-entropy is exactly minimizing \(D_{\mathrm{KL}}(p\,\|\,q)\) — pushing the model’s distribution toward the data’s. (And minimizing cross-entropy is equivalent to maximizing log-likelihood — the same objective from a statistician’s chair.)
import numpy as np
def bce(y, q, eps=1e-12):
q = np.clip(q, eps, 1 - eps) # guard against log(0)
return -(y*np.log(q) + (1-y)*np.log(1-q))
# A positive example (y=1) under increasingly confident predictions:
for q in [0.5, 0.7, 0.9, 0.99, 0.01]:
print(f"y=1, q={q:>4}: BCE = {bce(1, q):.4f}")y=1, q= 0.5: BCE = 0.6931
y=1, q= 0.7: BCE = 0.3567
y=1, q= 0.9: BCE = 0.1054
y=1, q=0.99: BCE = 0.0101
y=1, q=0.01: BCE = 4.6052The loss as a function of the predicted probability, for a positive example:
import matplotlib.pyplot as plt
q = np.linspace(0.001, 0.999, 200)
plt.figure(figsize=(5, 3))
plt.plot(q, bce(1, q), label="y = 1")
plt.plot(q, bce(0, q), label="y = 0", linestyle="--")
plt.xlabel("predicted probability q = P(y=1)")
plt.ylabel("binary cross-entropy")
plt.legend(); plt.grid(alpha=0.3); plt.tight_layout(); plt.show()
Cross-entropy is not legacy — it is the live training objective for essentially every classifier and language model in 2026.
| Concept here | What it becomes | Where you see it today |
|---|---|---|
| Binary cross-entropy | Categorical cross-entropy (softmax over \(K\) classes) | Every multi-class classifier; introduced later in the timeline |
| Cross-entropy over a vocabulary | Next-token prediction loss | The pretraining objective of every LLM |
| \(\exp(\text{cross-entropy})\) | Perplexity | The standard language-model metric |
| KL divergence | Regularizer / distance between distributions | VAEs, knowledge distillation, the KL penalty in RLHF, DPO |
| CE = \(-\log\)-likelihood | Maximum-likelihood estimation | The probabilistic justification for the whole setup |
→ Next: Activations & Non-linearity