40 Linear & Probabilistic Models

TL;DR

The simplest supervised models are linear in their parameters: logistic regression is a single sigmoid neuron trained by maximum likelihood, and naive Bayes is a generative classifier with a strong independence assumption. Both are the direct classical ancestors of the artificial neuron and the cross-entropy objective.

40.1 Logistic regression: the neuron, named earlier

A linear score passed through a sigmoid gives a class probability:

\[ p(y{=}1 \mid x) = \sigma(w^\top x + b), \qquad \sigma(z) = \frac{1}{1+e^{-z}}. \]

Training maximizes the likelihood of the labels — equivalently, minimizes the binary cross-entropy / log-loss:

\[ \mathcal{L} = -\sum_i \big[ y_i \log \hat p_i + (1-y_i)\log(1-\hat p_i)\big]. \]

This is exactly a single artificial neuron with a sigmoid output and the BCE loss from the foundations — the neural net is logistic regression stacked and made non-linear. The decision boundary is a hyperplane; its power comes entirely from the features it is given.

40.2 Naive Bayes: generative, with an independence shortcut

Rather than model \(p(y\mid x)\) directly, naive Bayes models the joint via Bayes’ rule and assumes features are conditionally independent given the class:

\[ p(y \mid x) \propto p(y)\prod_j p(x_j \mid y). \]

The independence assumption is “naive” and usually false, yet the classifier is fast, needs little data, and was the workhorse text-classifier (spam, sentiment) that statistical NLP relied on — the baseline that Seq2Seq and later LLMs displaced.

40.3 The unifying thread: maximum likelihood

Both models are fit by maximum likelihood, and minimizing negative log-likelihood is minimizing cross-entropy — the same principle that trains every network in this book. Logistic regression is discriminative (models \(p(y\mid x)\)); naive Bayes is generative (models \(p(x,y)\)). That discriminative/generative axis runs all the way up to BERT (masked, discriminative-ish) vs. GPT (generative).

Key takeaway

Linear/probabilistic models are not just baselines — they are the conceptual seed of the neural stack: logistic regression is one neuron, naive Bayes is maximum likelihood with an independence shortcut, and both are trained by the same cross-entropy that powers deep nets. They still win when data is scarce or interpretability is mandatory.

← Back to: Classical ML Overview