Ridge Regression (L2)
The L2 penalty's closed form, its Gaussian-prior view, and why it fixes conditioning.
On this page
Beginner: ordinary least squares (OLS) picks whichever coefficient vector makes the training errors as small as possible, full stop — it never asks whether those coefficients are themselves reasonable numbers. That's usually fine. But when some features move almost in lockstep with each other (highly correlated features), or when there are more features than data points to pin them down, OLS can respond by blowing up two (or more) coefficients to huge, opposite-signed values that happen to cancel out on the training set — a razor-thin, unstable balancing act that falls apart the moment new data arrives. Ridge regression fixes this with one small change: it adds a penalty for coefficients being large, in addition to the usual penalty for being wrong. The result is that every coefficient gets pulled toward zero by some amount — never landing exactly on zero, just shrunk — and that shrinkage is precisely what keeps the model from swinging wildly in response to noise.
Intermediate: this is the bias-variance trade-off from Module 1, made concrete. Shrinking every coefficient slightly means the fitted model can no longer match the training data quite as exactly as OLS does — it picks up a small amount of bias. But in exchange, the coefficients stop swinging wildly from one training sample to the next — the model's variance drops, often dramatically, especially when features are collinear or the feature count is close to (or exceeds) the sample count. The whole appeal of ridge is that this trade is usually extremely lopsided in your favor: a tiny amount of bias buys a large reduction in variance, and the net effect on error on new, unseen data is almost always positive. The knob controlling how much of this trade you take is a single hyperparameter, λ (lambda) — λ = 0 recovers plain OLS exactly, and larger λ shrinks every coefficient harder, all the way toward (never touching) the all-zeros vector as λ → ∞.
Advanced: the mechanism behind the shrinkage isn't just "add a penalty and hope" — it's a direct fix to a specific piece of linear algebra. OLS's closed form needs to invert X^TX, and that matrix is singular (not invertible at all) whenever features are exactly collinear or whenever there are more features than data points, and merely ill-conditioned (technically invertible, but numerically unstable — tiny changes in the data produce huge changes in the solution) whenever features are nearly collinear. Ridge replaces X^TX with X^TX + λI, and — as the derivation below shows precisely — adding λI shifts every eigenvalue of X^TX up by exactly λ. Whatever was zero or dangerously close to zero becomes safely positive for any λ > 0, so the matrix is now always invertible and the numerical instability is directly, structurally repaired — not just discouraged by a penalty term in a loss function, but made impossible by construction.
This lesson picks up immediately where Linear Regression (OLS) left off: same X, y, and squared-error objective, with one additional term bolted on. Everything derived here reduces exactly to the OLS result the moment λ = 0.
λ ≥ 0 is the ridge penalty strength, and I is the p × p identity matrix (p = number of features). At λ = 0 this is exactly the OLS normal equation from the previous lesson; every extra bit of structure below comes from that single added λI.
Part 1 — deriving the closed form. Start from the ridge objective, the usual sum of squared errors plus an L2 penalty on the coefficients:
Differentiate with respect to the vector θ using the same two matrix- calculus identities the OLS derivation relies on — ∂(θ^TA θ)/∂θ = 2Aθ for symmetric A, and the chain rule on the expanded quadratic:
Set the gradient to zero — the first-order condition for a minimum of this convex objective:
Divide by 2, move the data term to the right, and factor θ out on the left:
— the closed form stated above, and by inspection, setting λ = 0 collapses this exactly back to the OLS normal equation.
Why +λI guarantees invertibility. X^TX is always symmetric and positive semi-definite, so it has an eigendecomposition X^TX = VDV^T with an orthogonal matrix V and real, non- negative eigenvalues d₁, …, d_p ≥ 0 along the diagonal of D (this is exactly the eigenvalue machinery from the Linear Algebra chapter). "Positive semi-definite" rather than strictly positive definite is precisely the problem: whenever two or more features are exactly collinear, or whenever there are more features than data points (p > n), at least one eigenvalue is exactly zero, and X^TX is singular — has no inverse at all. Even when no eigenvalue is exactly zero, near-collinearity leaves the smallest eigenvalue very close to zero, and inverting a matrix with a near-zero eigenvalue is numerically treacherous — tiny noise in the data gets divided by that tiny number and amplified into huge swings in θ̂. Now look at what adding λI does to the same eigendecomposition:
Every eigenvalue is shifted up by exactly the same amount, λ: the new eigenvalues are d₁+λ, …, d_p+λ. For any λ > 0, even a previously-zero eigenvalue becomes 0 + λ = λ > 0 — strictly positive. The matrix X^TX + λI is therefore always invertible for λ > 0, regardless of how collinear the features are or how few data points there are relative to feature count. This is not a side effect of the penalty term discouraging large coefficients — it is a direct, guaranteed structural repair of the exact matrix that OLS needs to invert and sometimes can't.
Part 2 — the Gaussian-prior MAP equivalence. The Maximum Likelihood & MAP lesson (2.1.6) showed in general that choosing a loss is choosing a likelihood, and choosing a regularizer is choosing a prior. Here is that correspondence made completely explicit for ridge. Assume the same linear-Gaussian noise model behind OLS's MLE justification:
whose log-likelihood, summed over n independent observations, is (dropping nothing yet):
Now add a belief about θ before seeing any data — a zero-mean Gaussian prior, θ ~ N(0, τ²I), encoding "coefficients are probably small, with no preferred sign or direction," whose log-density is:
MAP maximizes log-likelihood plus log-prior together. Add the two expressions and drop every term that doesn't contain θ — those are additive constants as far as the maximization is concerned:
Flip the sign to turn "maximize" into "minimize," and multiply the whole expression through by 2σ² (a positive constant — multiplying an objective by a positive constant never changes its minimizer):
That is exactly the ridge objective stated above, with the regularization strength pinned to a concrete, interpretable ratio:
Read directly off this ratio: a noisy data-generating process (large σ²) or a strong prior belief that coefficients are small (small τ²) both push λ up — more shrinkage — and the two knobs trade off against each other in the single number a ridge implementation asks you to set. Ridge regression is MAP estimation of a linear model's weights under an independent, zero-mean Gaussian prior — not analogous to it, the identical optimization problem viewed from two directions, exactly the loss ⟺ likelihood, regularizer ⟺ prior correspondence 2.1.6 promised would show up again here.
Where this is used: the invertibility fix makes ridge the default, almost reflexive choice any time a design matrix is suspected of being rank-deficient or ill-conditioned — wide data (more features than rows), one-hot-encoded categorical variables with rare levels, polynomial or interaction features that are almost linear combinations of each other. The MAP reading is what justifies treating λ as a real, tunable belief about coefficient size rather than an arbitrary knob, and it's the exact mechanism generalized later by kernel ridge regression and by Tikhonov regularization in inverse problems more broadly (image deblurring, geophysical inversion) — all the same (A^TA + λI) structure protecting an otherwise unstable inversion.
Four features fitted on the same 40-point dataset; three of them (blue, green, amber) are deliberately built to be highly correlated. The dashed cursor and dots track the live ridge solution θ̂(λ) as the penalty grows from 0 (plain OLS) to a large value. Drag the slider yourself once the intro finishes, or hit replay.
A deliberately two-feature toy fit so coefficient space is the whole page. The faint rings are exact level sets of the OLS error surface around θ̂_OLS (red); the teal circle is the L2 ball whose radius equals ‖θ̂ ridge(λ)‖. The ridge solution (blue) always lands exactly where the highlighted error ellipse is tangent to that circle -- as λ grows the ball shrinks and drags the solution toward the origin, bending along the correlated-feature direction rather than moving straight toward it.
All three tabs implement the identical closed form derived above; the only difference is the linear-algebra plumbing. The library tab adds the two things a from-scratch implementation leaves for you to remember yourself: standardizing before fitting, and choosing λ (here, alpha) by cross-validation instead of a guess — both covered as pitfalls below.
#include <cmath>
#include <cstdio>
#include <vector>
#include <random>
#include <algorithm>
// Solve A x = b with Gaussian elimination and partial pivoting -- no linear-algebra library.
std::vector<double> solveLinearSystem(std::vector<std::vector<double>> A, std::vector<double> b) {
int n = static_cast<int>(b.size());
for (int col = 0; col < n; ++col) {
int pivot = col;
for (int r = col + 1; r < n; ++r)
if (std::fabs(A[r][col]) > std::fabs(A[pivot][col])) pivot = r;
std::swap(A[col], A[pivot]);
std::swap(b[col], b[pivot]);
for (int r = col + 1; r < n; ++r) {
double factor = A[r][col] / A[col][col];
for (int c = col; c < n; ++c) A[r][c] -= factor * A[col][c];
b[r] -= factor * b[col];
}
}
std::vector<double> x(n, 0.0);
for (int row = n - 1; row >= 0; --row) {
double s = b[row];
for (int c = row + 1; c < n; ++c) s -= A[row][c] * x[c];
x[row] = s / A[row][row];
}
return x;
}
std::vector<double> fitRidge(const std::vector<std::vector<double>>& X,
const std::vector<double>& y, double lambda) {
int n = static_cast<int>(X.size());
int p = static_cast<int>(X[0].size());
std::vector<std::vector<double>> XtX(p, std::vector<double>(p, 0.0));
std::vector<double> Xty(p, 0.0);
for (int i = 0; i < n; ++i) {
for (int a = 0; a < p; ++a) {
Xty[a] += X[i][a] * y[i];
for (int c = 0; c < p; ++c) XtX[a][c] += X[i][a] * X[i][c];
}
}
for (int d = 0; d < p; ++d) XtX[d][d] += lambda; // XtX + lambda * I
return solveLinearSystem(XtX, Xty);
}
int main() {
std::mt19937 rng(0);
std::normal_distribution<double> noise35(0.0, 0.35);
std::normal_distribution<double> noise40(0.0, 0.4);
std::normal_distribution<double> noise11(0.0, 1.1);
std::normal_distribution<double> noise05(0.0, 0.5);
const int n = 40, p = 4;
std::vector<std::vector<double>> X(n, std::vector<double>(p));
std::vector<double> y(n);
double trueTheta[4] = {2.4, -1.6, 1.1, 0.7};
for (int i = 0; i < n; ++i) {
double base = std::sin(i * 0.37) * 1.4 + std::cos(i * 0.21) * 0.6;
X[i][0] = base + noise35(rng);
X[i][1] = 0.9 * base + noise35(rng);
X[i][2] = -0.65 * base + noise40(rng);
X[i][3] = noise11(rng);
double s = 0.0;
for (int j = 0; j < p; ++j) s += trueTheta[j] * X[i][j];
y[i] = s + noise05(rng);
}
for (double lambda : {0.0, 1.0, 10.0, 100.0, 400.0}) {
std::vector<double> theta = fitRidge(X, y, lambda);
std::printf("lambda=%-7.1f theta=[%.3f, %.3f, %.3f, %.3f]\n",
lambda, theta[0], theta[1], theta[2], theta[3]);
}
return 0;
}- Genomics and quantitative genetics. A genome-wide association or genomic-prediction dataset routinely has tens of thousands of SNP markers as features and only hundreds or low thousands of individuals sampled —
p ≫ n, the exact regime whereX^TXis guaranteed singular. Genomic best linear unbiased prediction (GBLUP), a workhorse method for predicting traits from marker data, is mathematically ridge regression on the markers. - Chemometrics and spectroscopy. Near-infrared or Raman spectra are recorded as intensity at hundreds or thousands of adjacent wavelengths, which are necessarily highly correlated with their neighbors, on a few dozen or hundred lab samples. Ridge (and its relatives like partial least squares) is standard for regressing a chemical property (e.g. protein content) onto a full spectrum without the fit collapsing under collinearity.
- Econometrics with correlated macro indicators. GDP growth, unemployment, inflation, and interest rates all tend to move together across the business cycle. A regression trying to disentangle each indicator's individual effect on some outcome runs straight into near-collinear columns; ridge stabilizes the coefficient estimates instead of letting OLS assign implausibly large, offsetting weights to two indicators that are 90% redundant with each other.
- Image and signal restoration (Tikhonov regularization). Deblurring or deconvolving a signal means inverting a linear operator that is frequently ill-conditioned — small measurement noise, without regularization, gets amplified into huge, visually obvious artifacts in the reconstructed image. Adding a
λIterm to the operator before inverting it is the classical Tikhonov fix, and it is the literal same(A^TA + λI)structure as ridge. - Marketing mix modeling and ad attribution. Spend across TV, search, and social channels tends to rise and fall together with overall marketing budget and seasonality, making the channels' contributions to sales hard to separate with plain OLS. Ridge is a standard stabilizer in marketing mix models for exactly this reason.
- Weight decay in neural network training. The "weight decay" term added to a neural network's loss during training is an L2 penalty on the network's weights — the same Gaussian-prior MAP idea derived above, just applied to a much larger, non-linear model instead of a linear one.
- Forgetting to standardize features first. The penalty term
λ‖θ‖²punishes raw coefficient magnitude, and a coefficient's raw magnitude is partly just an artifact of the units its feature happens to be measured in — a feature recorded in millimeters needs a coefficient a thousand times smaller than the same information recorded in meters to produce the same prediction. Fit ridge on unstandardized features and a single sharedλends up applying wildly unequal effective penalties across features, punishing large-scale features far more than the underlying relationship warrants. Standardize (zero mean, unit variance) every feature before applying one sharedλacross all of them. - Expecting ridge to do feature selection. Ridge shrinks every coefficient toward zero, but — as both the diagram above and the SVD argument below show — it never sets one to exactly zero for any finite
λ. If the goal is an interpretable model with a genuinely sparse subset of active features, ridge is the wrong tool; that is exactly the gap the next lesson's Lasso (L1) penalty fills, by pushing coefficients all the way to exact zero instead of merely shrinking them. - Picking λ without cross-validation.
λ = 0gives back plain OLS (no protection against overfitting or instability);λ → ∞shrinks everything to zero (an intercept-only model, underfitting badly). The useful value sits somewhere in between, and where it sits depends entirely on the specific dataset — there's no universal default. Sweep a grid of candidateλvalues and choose by k-fold cross-validation (or use a built-in path/CV routine such asRidgeCV), rather than picking a value once and trusting it.
Going deeper
Ridge's shrinkage has an exact structural description via the SVD of X (the same decomposition from the Linear Algebra chapter). Write X = UDV^T with singular values d₁, …, d_p. Substituting into the closed form and simplifying (using X^TX = VD²V^T) gives the fitted values as:
Compare this to OLS's fitted values, ŷ_OLS = Σᵢ uᵢ(uᵢ^Ty) — the identical sum, but with every term multiplied by a shrinkage factor dᵢ²/(dᵢ²+λ) that is always strictly between 0 and 1. This factor is close to 1 (almost no shrinkage) for directions uᵢ where X has large singular value dᵢ — high-variance, well-supported directions in feature space — and close to 0 (aggressive shrinkage) for directions where dᵢ is small — exactly the low-variance, near-collinear directions responsible for OLS's instability in the first place. Ridge is not a blunt, uniform shrink of every coordinate by the same amount; it is a precisely graded shrink that leaves well-determined directions almost untouched and suppresses poorly-determined ones hardest — a smooth, soft cousin of the harder cutoff performed by discarding low-variance components entirely in principal component regression.
Two features in your dataset are almost perfectly collinear, so X^TX is (numerically) singular. Explain precisely why adding a ridge penalty with any λ > 0 fixes this, and describe what happens to the ridge coefficient vector at the two extremes λ → 0 and λ → ∞.
X^TX is symmetric positive semi-definite, so it has an eigendecomposition X^TX = VDV^T with eigenvalues d_1,…,d_p ≥ 0; near-perfect collinearity means at least one eigenvalue is zero or extremely close to zero, which is exactly what makes X^TX singular or numerically unstable to invert. Adding λI produces X^TX + λI = V(D+λI)V^T, whose eigenvalues are d_1+λ,…,d_p+λ — every eigenvalue shifted up by the same λ. For any λ > 0, even a previously-zero eigenvalue becomes λ > 0, so the matrix is guaranteed invertible; this is a structural fix to the matrix itself, not just a penalty that happens to discourage instability. At λ → 0, the ridge solution converges back to the (possibly unstable or undefined) OLS solution. At λ → ∞, the penalty term dominates the objective completely and every coefficient is driven toward 0, so θ̂_ridge → the zero vector (an intercept-only model) — bias approaching its maximum, variance approaching zero.
Ridge regression adds exactly one term to OLS's objective — λ‖θ‖² — and that one term does two things simultaneously, both derived precisely above. Structurally, it repairs the normal equations by shifting every eigenvalue of X^TX up by λ, guaranteeing invertibility even when features are collinear or p > n. Statistically, it is exactly MAP estimation under a Gaussian prior on the coefficients, with λ = σ²/τ² — trading a small, controlled amount of bias for a often much larger reduction in variance, the Module 1 bias-variance trade-off made concrete and tunable through a single hyperparameter. The SVD view sharpens this further: the trade isn't applied uniformly, it's graded by each direction's singular value, hitting the unstable, low-variance directions hardest and the well-supported ones least. What ridge does not do is produce an exactly sparse, feature-selecting model — every coefficient survives, just shrunk. That's precisely the gap the next lesson's Lasso (L1) penalty is built to close.