Lasso Regression (L1)
Why an L1 penalty produces exact zeros — the geometry, and the Laplace-prior view.
On this page
Beginner: ridge regression (section 2.3.2) shrinks every coefficient toward zero but never lands exactly on it. Lasso replaces ridge's ‖θ‖² penalty with ‖θ‖₁ = Σ|θⱼ| — and this small change in the penalty's shape has a big consequence: it pushes many coefficients to exactly zero, giving automatic feature selection bundled in with the regularization.
Intermediate: both methods shrink; only lasso zeros out. Ridge is the right choice when you believe every feature genuinely matters a little; lasso is the right choice when you believe only a handful of features matter and the rest are noise.
Advanced: the actual algorithm that solves the lasso objective — the coordinate-descent loop and its soft-thresholding update — was already derived in full, step by step, back in section 2.2.5. This lesson doesn't repeat that derivation; instead it explains something that lesson didn't cover: exactly why an L1 penalty produces sparsity in the first place, as a fact about geometry, plus the probabilistic (Laplace-prior) reading of the same penalty.
The per-coordinate solution is the soft-thresholding operator — fully derived in section 2.2.5 — applied inside a coordinate-descent loop.
Lasso's penalized objective has an equivalent CONSTRAINED form: minimize ‖y−Xθ‖² subject to ‖θ‖₁ ≤ t for some t matching λ (compare ridge's constrained form, ‖θ‖₂² ≤ t). Picture the unconstrained squared-error objective's level sets as ellipses centered at the OLS solution (section 2.3.1) — the constrained optimum is the smallest such ellipse that still touches the constraint region.
In 2D, the L1 region is a diamond, |θ₁|+|θ₂|≤t; the L2 region is a circle, θ₁²+θ₂²≤t. Every point on a circle's boundary is geometrically identical to every other — no direction is special — so for the growing ellipse to first touch the circle EXACTLY at an axis point (where θ₁=0) requires the ellipse's major axis to be aligned in one very particular, measure-zero way. Touching at a generic, non-axis point is overwhelmingly the typical outcome.
The diamond is different: it has four CORNERS, and a corner is not a single special direction but a whole RANGE of ellipse orientations and eccentricities for which the corner is still the first point touched — because at a corner, the diamond's boundary has a sharp kink, and any ellipse whose tangent line at that point falls anywhere within the wedge spanned by the diamond's two adjacent edges will touch there first. That's an open range of orientations, not one exact alignment — which is exactly why, for elongated or tilted ellipses (the correlated-feature case the diagram below shows), touching at a corner (one coefficient exactly zero) is the generic outcome, not a rare coincidence.
Now the probabilistic view. Give each θⱼ an independent Laplace prior, density p(θⱼ) ∝ exp(−|θⱼ|/b). Following the same MAP derivation pattern as Module 1's MLE/MAP lesson and the previous ridge lesson: the log-prior is −|θⱼ|/b + const, so the full MAP log-posterior (Gaussian likelihood plus Laplace log-prior) is:
Maximizing this is exactly minimizing ‖y−Xθ‖² + (2σ²/b)‖θ‖₁ — the lasso objective, with λ = 2σ²/b. A sharply-peaked Laplace prior at zero (small b) corresponds to a large λ and heavier shrinkage, exactly as intuition suggests.
Where this is used: the actual solving algorithm for both the geometric and probabilistic formulations is identical — the coordinate-descent + soft-thresholding loop from section 2.2.5.
Watch the red error contour grow outward from the unconstrained optimum. It first touches the violet diamond at a sharp corner (one coefficient exactly 0) but would touch a same-area green circle at an ordinary boundary point (neither coefficient 0).
Five features' fitted lasso coefficients plotted against λ, computed by rerunning the section 2.2.5 coordinate-descent algorithm at every point. Each colored tick on the zero line marks the λ where that coefficient first goes exactly flat -- including the deliberately irrelevant x₄, whose true coefficient really is 0. Drag the slider yourself once the intro finishes, or hit replay.
#include <cmath>
#include <iostream>
#include <vector>
double softThreshold(double theta, double lam) {
double mag = std::abs(theta) - lam;
return mag > 0 ? (theta > 0 ? 1.0 : -1.0) * mag : 0.0;
}
int main() {
// Small hand-built example: 2 relevant features, 1 irrelevant one.
std::vector<std::vector<double>> X = {{1, 0, 2}, {0, 1, -1}, {1, 1, 0}, {2, -1, 1}};
std::vector<double> y = {2, -1.5, 0.5, 4.5};
int n = X.size(), p = X[0].size();
std::vector<double> theta(p, 0.0), colSq(p, 0.0);
for (int j = 0; j < p; ++j)
for (int i = 0; i < n; ++i) colSq[j] += X[i][j] * X[i][j];
double lambda = 0.8;
for (int iter = 0; iter < 200; ++iter) {
for (int j = 0; j < p; ++j) {
double b = 0.0;
for (int i = 0; i < n; ++i) {
double pred = 0.0;
for (int k = 0; k < p; ++k) if (k != j) pred += X[i][k] * theta[k];
b += X[i][j] * (y[i] - pred);
}
theta[j] = softThreshold(b, lambda) / colSq[j];
}
}
for (double t : theta) std::cout << t << " ";
std::cout << "\n";
return 0;
}- Genomics — selecting a handful of genes out of tens of thousands of candidates that actually predict an outcome.
- Sparse signal recovery — reconstructing a signal known to have very few non-zero components from limited measurements.
- Text classification — huge sparse bag-of-words feature vectors, where only a small vocabulary subset is actually predictive for a given task.
- Regulated industries (finance, healthcare) — models whose "which features matter" list needs to be short and auditable, not a dense weighted sum of hundreds of inputs.
- Exploratory feature screening — running lasso as a fast first pass to narrow down a large feature set before fitting a more complex downstream model.
- Lasso's instability among groups of highly correlated features — it tends to arbitrarily pick just one from the group and zero out the rest, rather than spreading credit sensibly (directly motivating elastic net, the next lesson).
- Forgetting to standardize features before applying one shared
λ— an unscaled feature gets an effectively different penalty than the rest. - Treating lasso's selected subset as a reliable "ground truth" — it's one of possibly several similarly-good sparse solutions, especially when features are correlated.
Going deeper
Lasso is only guaranteed to recover the TRUE sparse support (not just A sparse solution) under a technical condition on X known as the irrepresentable condition — informally, the irrelevant features can't be too strongly correlated with the relevant ones. When predictors are highly correlated, this condition can fail, and lasso's selected support can genuinely diverge from the true one even with plenty of data — a real limitation worth knowing about rather than assuming lasso always "finds the right answer."
Ridge and lasso both shrink coefficients toward zero. Why does only lasso produce EXACT zeros?
Geometrically, ridge's constraint region (a circle/sphere) has no special points — the growing error ellipse touches it at a generic boundary point where no coordinate is zero. Lasso's constraint region (a diamond/cross-polytope) has sharp corners exactly on the coordinate axes, and a whole range of ellipse shapes and orientations touch the region first at one of those corners — which is precisely where one or more coordinates equal zero. It's a consequence of the L1 ball's shape, not an artifact of how the optimization is solved.
Lasso's sparsity isn't a numerical accident of coordinate descent — it's a direct geometric consequence of the L1 ball having corners, and an equally direct probabilistic consequence of a Laplace prior's sharp peak at zero. The next lesson, Elastic Net, keeps this sparsity while fixing the one real weakness just flagged: instability among correlated features.