Elastic Net
Blending L1 and L2 penalties, and the modified soft-thresholding update it needs.
On this page
Beginner: ridge regression (2.3.2) shrinks every coefficient a little, but almost never all the way to zero. Lasso (2.3.3) can zero coefficients out entirely, which is great for automatic feature selection. Elastic net just blends the two penalties together with a dial, so you get a bit of both behaviors instead of having to commit to one extreme or the other.
Intermediate: the dial is a mixing parameter α ∈ [0, 1]. Set α = 1 and the penalty is pure L1 — plain Lasso. Set α = 0 and it's pure L2 — plain ridge. Anything in between blends the two. The reason this is worth a whole lesson, rather than being a footnote on Lasso, is a concrete failure mode Lasso has: when two or more features are strongly correlated with each other, Lasso's L1 penalty tends to pick just one of them somewhat arbitrarily and zero out the rest — and which one it picks can flip between two very similar datasets or even two different runs on resampled data. That's a real instability, not a hypothetical one. Elastic net's added L2 component fixes it directly: instead of arbitrarily crowning one correlated feature the winner, it tends to keep correlated features' coefficients close to each other in size, splitting the credit between them roughly evenly rather than gambling it all on one.
Advanced: there are two hyperparameters doing distinct jobs, and it's worth keeping them mentally separate: λ controls overall regularization strength (how hard the penalty pushes, period), while α controls the mixture of L1-vs-L2 character within that fixed total strength. This "grouping effect" — nearly-identical coefficients for nearly-identical (highly correlated) features — is not just a qualitative impression; Zou & Hastie's original 2005 paper introducing elastic net proves a formal bound on how far apart two correlated features' fitted coefficients can be, and that bound shrinks toward zero as the correlation between the features approaches 1 and as the L2 share of the penalty grows. The derivation below makes the mechanism completely explicit: it's one extra additive term in the denominator of the coordinate update, contributed entirely by the L2 piece.
α = 1 recovers Lasso exactly; α = 0 recovers ridge exactly. λ sets the total regularization strength, and α sets how that total is split between the sparsity-inducing L1 piece and the grouping-inducing L2 piece.
As in the Lasso coordinate-descent derivation (2.2.5), it's convenient to work with an equivalent ½‖y − Xθ‖² scaling of the squared-error term — multiplying the whole objective by a constant doesn't change which θ minimizes it, it just rescales λ by that same constant. Fix every coefficient except θⱼ and isolate everything that depends on it. The 1D sub-problem becomes minimizing:
exactly as before, a = Σᵢ Xᵢⱼ² (column j's squared norm) and b = Σᵢ Xᵢⱼ(yᵢ − Σₖ≠ⱼ Xᵢₖθₖ) (column j's correlation with the current partial residual). The only difference from the pure-Lasso sub-problem is the last term, and it's smooth everywhere — it has no corner at zero the way λα|θⱼ| does. That means it can be folded straight into the other smooth, quadratic term instead of needing its own case analysis:
Substituting that back in, the sub-problem collapses to:
which is identical in shape to plain Lasso's 1D sub-problem, with a replaced by ã and λ replaced by λα in the penalty term only. So the exact same three-case argument applies, just carried through with those two substitutions:
- Case θⱼ > 0: the penalty is smooth here (
+λαθⱼ); differentiateãθⱼ − b + λα, set to zero, givingθⱼ = (b − λα)/ã— valid only whenb > λα. - Case θⱼ < 0: symmetric, giving
θⱼ = (b + λα)/ã— valid only whenb < −λα. - Case θⱼ = 0: valid exactly when some value in the L1 subgradient set
[−λα, λα]can zero out the total subgradient−b + [−λα, λα], i.e. whenever|b| ≤ λα.
These three cases combine into one closed form — a rescaled soft-thresholding update:
Reduction to Lasso (α = 1): the denominator's extra term 2λ(1−α) vanishes, and the numerator's threshold is λα = λ, so the whole expression becomes soft(b, λ)/a — exactly the Lasso update from 2.2.5, with no approximation involved.
Reduction to ridge (α = 0): the L1 threshold becomes λα = 0, and soft(b, 0) = b (soft-thresholding with a zero threshold never clips anything), so the update becomes θⱼ* = b/(a + 2λ) — precisely ridge regression's single-coordinate closed form: the same numerator as OLS's normal equations, with 2λ added directly onto the denominator instead of any thresholding.
Where this is used: that 2λ(1−α) term sitting in the denominator is the entire mechanism behind fixing Lasso's correlated-feature instability. It inflates the effective denominator for every coefficient by the same fixed amount regardless of which feature currently "wins" the soft-thresholding competition, which caps how much larger any one correlated feature's coefficient can grow relative to its peers — the larger (1−α) is, the tighter that cap, and the more evenly credit gets split across a correlated group instead of concentrating arbitrarily on whichever one happened to look marginally better to the optimizer first.
On load, α sweeps continuously from 1 down to 0 and back to 1 — watch the sharp-cornered diamond melt into a smooth circle and back. The corners are exactly where sparsity (a coefficient landing at 0) happens; rounding them off is exactly what the L2 share buys you. Drag the slider to park α wherever you like.
Three genuinely correlated features (ρ ≈ 0.999) and two genuinely irrelevant ones, fit at a fixed λ = 6 while α sweeps from 1 (pure Lasso) to 0 (pure ridge). At α = 1 Lasso crowns one of the three correlated features almost arbitrarily and starves the other two; watch the bars converge to nearly equal heights as α drops, while the two gray, genuinely-irrelevant bars stay pinned near zero the entire time. The live standard-deviation readout is the same quantity the Zou & Hastie grouping bound (Advanced paragraph, and the code example's printed std) is about.
All three tabs build the same synthetic setup: three "co-regulated gene" columns sharing one latent factor (correlation above 0.95 with each other), plus seven columns of pure, genuinely irrelevant noise. Running the derived update with α = 1 reproduces plain Lasso's behavior — arbitrarily concentrating weight on one of the three correlated columns — while α = 0.3 visibly narrows the spread of coefficients across that same group.
#include <cmath>
#include <iostream>
#include <random>
#include <vector>
double softThreshold(double x, double thresh) {
double mag = std::abs(x) - thresh;
return mag > 0 ? (x > 0 ? 1.0 : -1.0) * mag : 0.0;
}
std::vector<double> elasticNetCoordinateDescent(
const std::vector<std::vector<double>>& X,
const std::vector<double>& y,
double lam,
double alphaMix,
int iters = 300
) {
int n = X.size(), p = X[0].size();
std::vector<double> theta(p, 0.0);
std::vector<double> colSq(p, 0.0);
for (int j = 0; j < p; ++j)
for (int i = 0; i < n; ++i) colSq[j] += X[i][j] * X[i][j];
for (int it = 0; it < iters; ++it) {
for (int j = 0; j < p; ++j) {
double b = 0.0;
for (int i = 0; i < n; ++i) {
double pred = 0.0;
for (int k = 0; k < p; ++k) if (k != j) pred += X[i][k] * theta[k];
b += X[i][j] * (y[i] - pred);
}
double aTilde = colSq[j] + 2.0 * lam * (1.0 - alphaMix);
theta[j] = softThreshold(b, lam * alphaMix) / aTilde;
}
}
return theta;
}
int main() {
std::mt19937 rng(0);
std::normal_distribution<double> noise(0.0, 1.0);
const int n = 150;
// Same setup as the Python version: three correlated "gene" columns sharing a
// latent factor z, plus seven genuinely irrelevant noise columns.
std::vector<double> z(n);
for (int i = 0; i < n; ++i) z[i] = noise(rng);
std::vector<std::vector<double>> X(n, std::vector<double>(10));
std::vector<double> y(n);
std::normal_distribution<double> smallNoise(0.0, 0.05);
for (int i = 0; i < n; ++i) {
for (int c = 0; c < 3; ++c) X[i][c] = z[i] + smallNoise(rng);
for (int c = 3; c < 10; ++c) X[i][c] = noise(rng);
y[i] = 4.0 * z[i] + 0.3 * noise(rng);
}
auto lassoLike = elasticNetCoordinateDescent(X, y, 6.0, 1.0);
auto elastic = elasticNetCoordinateDescent(X, y, 6.0, 0.3);
std::cout << "Lasso-equivalent (alpha=1): ";
for (double v : lassoLike) std::cout << v << " ";
std::cout << "\nElastic net (alpha=0.3): ";
for (double v : elastic) std::cout << v << " ";
std::cout << "\n";
return 0;
}- Genomics and bioinformatics — groups of genes that are co-regulated (turned on and off together as part of the same biological pathway) are, by definition, highly correlated across samples. Elastic net is a standard tool here specifically because it keeps a whole co-regulated group visible in the fitted model with similar weights, instead of Lasso's tendency to arbitrarily keep one gene from the group and silently drop the rest.
- Any high-dimensional (p ≫ n) setting with correlated features — microarray data, text classification with large, overlapping vocabularies, or wide tabular datasets with redundant engineered features. Zou & Hastie's original 2005 paper specifically motivated elastic net as a fix for Lasso being provably unstable exactly in this "more features than samples, with correlation" regime.
- Production ML pipelines as a safe default — when it isn't obvious in advance whether a dataset's true structure favors L1's aggressive sparsity or L2's gentler, non-zeroing shrinkage, elastic net with a moderate
αhedges between the two, which is why many teams reach for it by default in a regularized-linear-model pipeline rather than committing to pure Lasso or pure ridge upfront. - Neuroimaging (e.g. fMRI-based decoding) — neighboring voxels are almost always strongly spatially correlated with each other, and elastic net is widely used in that literature for exactly the grouping-effect reason: it tends to recover clusters of jointly-informative voxels rather than an arbitrary single voxel per cluster.
- Marketing mix / attribution modeling — ad spend across related channels (e.g. multiple social platforms run as part of one coordinated campaign) tends to move together, so a Lasso fit can arbitrarily credit one platform and zero out a genuinely contributing one; elastic net spreads attribution more evenly across the correlated group.
- Finance and econometrics — macroeconomic factors (interest rates, inflation expectations, various market indices) are frequently correlated with one another; elastic net is a common choice when the goal is a stable, reproducible factor model rather than a fit that reshuffles which factor "wins" every time the sample window shifts slightly.
- Tuning α and λ one at a time. They interact — the best λ for a given α is generally not the best λ for a different α. Cross-validate over a genuine 2D grid of both (e.g.
ElasticNetCV's combined search), not by fixing one and sweeping only the other, or the result can look far worse than either extreme purely from an unlucky search path. - Assuming elastic net always beats pure Lasso or pure ridge. It doesn't, unconditionally. On some datasets — say, one with a small number of truly independent, individually strong predictors and no meaningful correlation structure at all — pure Lasso's sparsity genuinely wins outright, and the L2 share only adds unnecessary bias. Elastic net widens the space of models you can reach; it doesn't guarantee the best one in that space is strictly interior.
- Forgetting to standardize features. Exactly as with ridge and Lasso individually, both penalty terms compare coefficient magnitudes directly against each other and against λ. A feature on a wildly different scale than the rest gets penalized unevenly relative to its actual predictive contribution unless every column is standardized first — this requirement doesn't go away just because two penalties are blended instead of one.
Going deeper
The update derived above — solved directly from the stated objective, with no further adjustment — is what Zou & Hastie's original 2005 paper calls the naive elastic net, and they flag it as exactly that: naive, in the sense that it double-shrinks. The L1 and L2 penalties are each individually shrinking the coefficients toward zero, and stacking them raw combines both shrinkage effects without any offsetting benefit — the fit ends up more biased than either penalty alone would produce at a comparably-tuned strength, without a matching reduction in variance to show for it. Their fix is a simple linear rescaling: multiply the naive solution by a factor of 1 + λ(1 − α) (in their original parameterization, 1 + λ₂) to undo exactly the extra shrinkage contributed by the L2 term's presence in the denominator derived above, while leaving the grouping and sparsity behavior intact. This is a genuine, citable implementation subtlety: some solvers apply this rescaling as part of fitting, and it is worth checking a given library's documentation for whether its returned coefficients are the raw (naive) solve or the bias-corrected one before comparing magnitudes across tools or against a from-scratch implementation like the ones above.
Two features are correlated at ρ ≈ 0.99 and both are genuinely predictive. A strongly-regularized Lasso fit keeps one of them and drives the other to exactly zero. Using the rescaled update θⱼ* = soft(b, λα) / (a + 2λ(1-α)), explain what changes as α is lowered from 1, and which specific term is responsible.
Lowering alpha below 1 does two things to the update at once. First, the L1 threshold lambda*alpha shrinks, so it takes a smaller |b| to escape the zero region -- less aggressive zeroing, on its own. Second, and more importantly for the grouping behavior, the denominator gains the extra term 2*lambda*(1-alpha), which grows as alpha drops. That term inflates the effective denominator by the same fixed amount for every coefficient regardless of which feature's b happens to be largest, which caps how much larger any one correlated feature's coefficient can end up relative to its correlated peer -- instead of one feature's b winning the soft-thresholding competition outright and the other landing at exactly zero, both features end up with similar, non-zero coefficients. The larger (1-alpha) is, the tighter that cap, and the more evenly the two correlated features share the credit.
Elastic net is not a new derivation from scratch — it's the exact same coordinate-descent machinery from the Lasso lesson, with one smooth extra term folded into the quadratic part before soft-thresholding is applied, producing a single rescaled formula that collapses exactly onto Lasso at α = 1 and exactly onto ridge at α = 0. The one new term in the denominator, 2λ(1 − α), is the entire fix for the specific instability the previous lesson flagged: it stops correlated features from having their coefficients decided by an arbitrary tie-break, and spreads credit across the whole correlated group instead.