Foundations of Learning
What "learning" means mathematically, and the ideas every model shares.
On this page
This chapter covers classical machine learning end to end — every model family a working ML engineer or researcher is expected to know, derived from first principles rather than presented as a black box. It's cross-checked against Stanford's CS229, The Elements of Statistical Learning, and scikit-learn's full algorithm catalog, organized into 18 modules from foundations through interpretability.
It builds directly on the Linear Algebra for ML chapter — eigendecompositions, gradients, and matrix calculus from that chapter are used here without re-derivation, so it's worth having worked through that chapter first (or at least keeping it open for reference).
Every numbered topic below a module (like 2.1.1, 2.1.2, …) follows the same shape, going noticeably deeper than a typical course precisely because the goal here is to leave nothing as an unexplained black box:
- In plain English — the idea explained at beginner, intermediate, and advanced depth, with no formulas required to follow along.
- Formula & Derivation — the precise mathematical statement, proved or derived from scratch, with a "where this is used" note tying it to a real system.
- Diagram — a live, interactive diagram built to be readable at a glance for a beginner, but with enough real numbers and controls exposed that an expert can use it to actually check their intuition.
- Implemented three ways — the core algorithm coded by hand in Python (readable, minimal), then again from scratch in C++ (so you see it without any language hiding the mechanics), and finally how it's actually called in production via scikit-learn (or the standard library for that topic) — so you see the same idea at three levels of abstraction back to back.
- Real-world examples, pitfalls, and going deeper — where the idea shows up in production ML systems, what trips people up in practice, and an optional deeper dive.
Before any specific model — linear regression, decision trees, neural networks — there's a small set of ideas that every one of them is a special case of. This module builds exactly those ideas, so that every later module can say "this is just ERM with a particular hypothesis class and loss" instead of re-deriving why models are trained the way they're trained each time.
Empirical risk minimization — choose the function f, out of some allowed hypothesis class ℋ, that minimizes the average loss on the training data. Every model in this chapter fills in ℋ and L differently; the minimization principle itself never changes.
- 2.1.1 Learning Paradigms — supervised, unsupervised, semi-supervised, self-supervised, and reinforcement learning: what data each one assumes, and what "learning" even means when there are no labels.
- 2.1.2 Statistical Decision Theory — where loss functions actually come from, and the Bayes-optimal predictor that every model is trying to approximate.
- 2.1.3 Bias–Variance Trade-off — decomposing a model's expected error into three independent, additive pieces, and why you can't drive all three to zero at once.
- 2.1.4 Overfitting, Underfitting & Capacity — the practical, measurable symptoms of the bias-variance trade-off, including the modern "double descent" wrinkle in the classical story.
- 2.1.5 Learning Theory — the theoretical guarantees (PAC learning, VC dimension) behind "if it works on training data, will it work on new data?", and why no algorithm can be best at everything (No-Free-Lunch).
- 2.1.6 Maximum Likelihood & MAP — the single inference principle that, it turns out, almost every loss function in this entire chapter is secretly a special case of.
- Treating this module as throat-clearing to skip past to "the real models" — nearly every design decision in later modules (why squared loss for regression, why cross-entropy for classification, why regularization helps, why a bigger model isn't always better) is a direct consequence of an idea introduced here.
- Reading the math once and moving on — these six ideas are the ones you'll want to revisit after a few later modules, once you've seen them in action inside a specific model. It's normal for it to click more the second time.
Going deeper
If you already know these six topics cold, this module is still worth skimming for notation — later modules refer back to it constantly (e.g. "this is ERM with hinge loss," "this is the MAP estimate under a Laplace prior") rather than re-explaining the underlying principle each time.
Start with 2.1.1, Learning Paradigms, and move through the module in order — each topic sets up vocabulary the next one uses directly, ending with MLE/MAP, which is the thread that ties almost every model in every later module together.