Regression
Every classical regression method, from OLS to isotonic regression.
On this page
Regression — predicting a continuous number — is the single richest family of models in classical ML, precisely because there are so many different, principled answers to "what could possibly go wrong with the simplest version, and how do you fix it." This module works through that entire lineage: starting from ordinary least squares, then adding a penalty to fix instability (ridge), a different penalty to also get sparsity (lasso), a blend of both (elastic net), a path-based view of the whole regularization spectrum (LARS), non-linear basis expansions (polynomials, splines, GAMs), the exponential-family generalization that covers classification as a special case (GLMs), robustness to corrupted data (Huber/RANSAC), full conditional distributions instead of just a mean (quantile regression), regression on latent directions instead of raw features (PCR/PLS), greedy and grouped sparsity (OMP, group lasso), fully non-parametric local fits (LOESS), and finally regression under a pure ordering constraint (isotonic regression).
Every one of these is built directly on tools from the first two modules: the normal equations reuse projections and matrix inverses (the Linear Algebra chapter), penalties reuse the MAP/prior framing (Module 1), and the harder objectives reuse coordinate descent, proximal operators, and IRLS-as-Newton's-method (Module 2). This module is where those tools finally get pointed at real, named, widely-used models.
Every method in this module fits into the same regularized empirical risk template:
A loss L measuring fit, and a penalty R controlling complexity, weighted by λ. OLS is this template with R = 0. Ridge sets R = ‖θ‖². Lasso sets R = ‖θ‖₁. Elastic net blends both. Robust regression changes L instead of R. Quantile regression changes L to the pinball loss. GAMs and splines change what Xθ even means (a sum of smooth functions instead of a single linear combination). Almost every named method in this module is this one template with a specific, motivated choice of L and R.
- 2.3.1 Linear Regression (OLS) — the baseline: squared loss, the normal equations, and the Gauss-Markov theorem's guarantee.
- 2.3.2 Ridge Regression (L2) — a closed-form fix for instability and collinearity, and its Gaussian-prior (MAP) reading.
- 2.3.3 Lasso Regression (L1) — the penalty that produces exact zeros, and why that's a geometric fact about the L1 ball.
- 2.3.4 Elastic Net — blending both penalties to get sparsity without lasso's instability among correlated features.
- 2.3.5 Least Angle Regression (LARS) — the entire regularization path, computed almost as cheaply as one OLS fit.
- 2.3.6 Polynomial & Basis-Function Regression — curves fit by a method that's still, underneath, exactly linear regression.
- 2.3.7 Splines & Smoothing Splines — piecewise polynomials with a roughness penalty instead of a fixed degree.
- 2.3.8 Generalized Additive Models (GAM) — a sum of smooth functions per feature, fit by backfitting.
- 2.3.9 Generalized Linear Models (GLM) — the exponential-family generalization that includes logistic regression as a special case.
- 2.3.10 Robust Regression — fits that don't get dragged around by a handful of corrupted points.
- 2.3.11 Quantile Regression — predicting any conditional quantile, not just the mean.
- 2.3.12 PCR & PLS — regressing on a small number of latent directions instead of every raw feature.
- 2.3.13 OMP & Grouped Sparsity — greedy sparse recovery, and sparsity over whole groups of coefficients at once.
- 2.3.14 Locally Weighted Regression (LOESS) — a fully non-parametric fit built from many tiny local regressions.
- 2.3.15 Isotonic Regression — the best possible fit under nothing but a monotonicity constraint.
- Treating this module as a menu to pick exactly one method from — in practice, regularization strength, basis choice, and robustness are usually independent decisions, and production pipelines often combine several of these ideas at once (e.g. a GAM with elastic-net-penalized smooth terms).
- Assuming a more flexible method (splines, LOESS, GAMs) is always better than plain OLS — every one of these buys flexibility at the cost of interpretability, statistical efficiency, or both, and the bias-variance trade-off from Module 1 applies exactly as much here as anywhere else.
Going deeper
Several methods in this module deliberately revisit exactly the same result from a different angle — OLS's normal equations reappear inside ridge, lasso, GAMs, and GLMs; the coordinate-descent machinery from Module 2 reappears in lasso, elastic net, and GAM backfitting. That repetition is intentional: seeing the same handful of primitives combine in different ways is what makes the whole space of regression methods feel like one coherent story instead of fifteen unrelated tricks.
Start with 2.3.1, Linear Regression (OLS) — literally every other lesson in this module either adds a penalty to it, changes its loss function, changes what "linear" means, or generalizes its distributional assumption. Understanding OLS completely is what makes the other fourteen lessons feel like small, well-motivated edits rather than fifteen separate things to memorize.