39  Classical ML: Context for the Neural Timeline

← Back to: Home

TipTL;DR

This book’s spine is neural architectures, but they did not arrive into a vacuum — they displaced a mature classical-ML stack, and on some problems have not displaced it at all. This section is the background the main notes assume or contrast against: the linear/probabilistic models the artificial neuron generalizes, the SVMs that ruled 1995–2012, the tree ensembles still SOTA on tabular data in 2026, and the unsupervised methods behind “representation learning.”

39.1 Why classical ML belongs as context, not spine

The neural narrative is cleaner than the history. Three honest reasons to keep classical ML in view:

1. Continuity. Logistic regression is a single artificial neuron with a sigmoid; cross-entropy training of a net is maximum likelihood, the same principle behind naive Bayes. The foundations chapters are classical ML wearing neural notation.
2. The 2012 inflection. SVMs with hand-engineered features were the vision champion until AlexNet (Krizhevsky et al., 2012). Knowing what was displaced is what makes the deep-learning turn legible — see the historical chapter.
3. Where neural nets still lose. Gradient-boosted trees remain the default for tabular/structured data. “Deep learning won” is true for perception and language, not universally.

39.2 The supervised / unsupervised split

	Learns from	Goal	This section
Supervised	labelled \((x, y)\)	predict \(y\)	linear/probabilistic, SVM, trees
Unsupervised	unlabelled \(x\)	find structure	k-means, PCA

The neural era blurs this line — self-supervised pre-training (BERT, GPT) manufactures labels from raw text — but the classical split is the right starting frame.

39.3 Where each approach still wins

Method	Still the right tool when…
Logistic regression / naive Bayes	data is scarce, interpretability is required, baseline needed fast
SVM + kernel	small/medium data, clear margin, non-linear but low-dimensional
Gradient-boosted trees	tabular/structured features, mixed types, the production default
k-means / PCA	exploratory analysis, compression, visualization, preprocessing

39.4 Map of this section

• Linear & Probabilistic Models — logistic regression, naive Bayes; the MLE thread that ties to information theory.
• SVM & Kernels — max-margin classifiers and the kernel trick; the pre-2012 champion.
• Trees & Ensembles — decision trees, random forests, gradient boosting; the tabular SOTA.
• Unsupervised Methods — k-means and PCA; the classical ancestors of representation learning.

← Back to: Home

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 1097–1105.