Principal Components Regression & Partial Least Squares
Regressing on latent directions — variance-only versus covariance-aware ones.
On this page
Beginner: when you have many features that are highly correlated with each other, regressing on all of them directly can be numerically unstable — exactly the problem that motivated adding a penalty in ridge regression (2.3.2). A different fix is to not use the raw features at all: first compress them down into a small handful of new, derived directions, and regress on those instead. The whole question this lesson answers is how you should choose those directions.
Intermediate: Principal Components Regression (PCR) answers in two completely separate steps. Step one looks only at the feature matrix X — run PCA, covered in the Linear Algebra for ML chapter (section 1.10, Singular Value Decomposition), to find X's directions of largest variance, keep only the top k of them, and project every data point onto that small subspace. Step two is ordinary least squares of y on those k projected coordinates. Notice what's missing from step one: y never appears in it. The directions are chosen purely by looking at how X spreads out, exactly as the Linear Algebra chapter's "Why It Matters" lesson frames PCA in general — find where the data varies the most, full stop, with no notion yet of what you're trying to predict.
Advanced: Partial Least Squares (PLS) changes exactly one design decision, but the consequences are large. Instead of picking directions by how much variance they explain in X alone, PLS picks directions by how much covariance they capture with y. Its very first latent direction is chosen with the actual regression target already in the room — not discovered afterward by regressing on whatever PCA happened to hand it. The practical payoff is that PLS very often needs dramatically fewer latent components than PCR to reach comparable predictive accuracy, precisely because its first component is already aimed at y, rather than aimed at X's own internal spread and hoping that spread happens to line up with what you actually care about.
Worked intuition. Picture 200 highly correlated sensor readings where the dominant source of variation is something boring and irrelevant — say, ambient temperature drift affecting every sensor together — while the thing you actually want to predict depends on a much smaller, subtler pattern buried underneath that drift. PCR's first principal component will latch straight onto the big, boring temperature-drift direction, because that's where most of the variance in X lives, regardless of whether it has anything to do with y. PLS's first component, having looked at y from the start, goes straight for the subtler pattern instead. The diagram below builds exactly this scenario and lets you watch the two directions land in different places.
PCR, given the SVD X = UΣV^T:
PLS, its first latent direction:
PCR's directions come from an eigendecomposition of X alone (maximizing variance, a property of X by itself); PLS's first direction maximizes squared covariance with y — a property of X and y together. That single word swap, variance versus covariance, is the entire difference between the two methods.
Part 1 — PCR is exactly PCA followed by OLS, made explicit. Start from the SVD of the (column-centered) feature matrix:
Let V_k be the first k columns of V — the top k principal directions — and project:
Because V's columns are orthonormal, V_k^TV = [I_k \; 0], so:
A diagonal Z^TZ means the OLS step on the projected features has a trivial closed form, one coordinate at a time:
Reading that last line right to left: PCR's fitted values are literally "project X onto its top-k PCA directions, then do plain OLS in that reduced coordinate system." There is no separate "PCR algorithm" beyond those two familiar steps chained together — the implicit assumption baked into every PCR fit is that whatever signal predicts y must live inside span(V_k), the subspace of X's k largest-variance directions.
That assumption is exactly where PCR can fail, and the failure mode is precise, not vague: suppose the direction that happens to correlate most strongly with y is only X's 5th-largest-variance principal component, and you chose k = 2. Step one discarded that 5th component before y was ever consulted — it wasn't "down-weighted" or "penalized," it was thrown away entirely, on variance grounds alone. No amount of OLS fitting in step two can recover a direction that no longer exists in Z. PCR with a small k is blind to exactly this kind of high-relevance-but-low-variance signal, by construction, not by bad luck.
Part 2 — PLS's first direction, derived exactly. The objective is:
Write c = X^Ty, a fixed vector once the data is fixed, so the objective is (w^Tc)^2. Form the Lagrangian for this equality-constrained problem, using exactly the machinery of Module 2's Constrained Optimization & Duality lesson (2.2.6):
Differentiate with respect to w and set to zero:
The right-hand side is a scalar multiple of c — so the stationarity condition alone forces w to be parallel to c, with the unit-norm constraint then pinning down exactly which multiple:
The same conclusion falls out even more directly from Cauchy-Schwarz: (w^Tc)^2 \le \|w\|^2\|c\|^2 = \|c\|^2 for any unit w, with equality exactly when w and c point the same way — so w \propto c is not just a critical point, it's the global maximizer.
Compare the two first directions side by side: PCA/PCR's first direction is the top eigenvector of X^TX — a property of X alone, with y nowhere in the formula. PLS's first direction is literally X^Ty, normalized — the vector of feature-target covariances itself, y baked in from the very first step.
Where this is used: this is precisely why PLS so often needs far fewer latent components than PCR to reach the same predictive accuracy — its first direction is already pointed at the target, rather than at whatever happens to dominate X's own spread. The cost is that PLS's components can no longer be described as "the directions of maximum variance in X," the interpretation PCA/PCR components keep — PLS trades that interpretability for predictive efficiency.
Color encodes y (blue = low, red = high). The point cloud is built so y varies mostly along its SHORT axis. Watch the black PCA arrow lock onto the long axis of the cloud (maximum X-variance, ignoring color entirely) while the violet PLS arrow tilts toward wherever the color actually changes fastest (maximum covariance with y).
A sharper version of the 'background vs relevant' setup: 6 features, 4 independent high-variance factors y never depends on, and 2 low-variance features that load on the one factor it does. Drag k and compare training R² — PCR (black) stays flat while it burns through the four nuisance directions, then jumps once it finally reaches the signal; PLS (violet) heads for the signal from k=1 because its direction is chosen with y already in view. Both converge to the same full-rank OLS R², the dashed line, exactly as the ExpertNote describes.
All three tabs build the same synthetic data: a "background" factor with large variance that y does not depend on, and a smaller "relevant" factor that it does — the concrete version of the derivation's warning about a high-variance-but-irrelevant direction. In every tab, PLS's single covariance-seeking component matches or beats PCR's top variance-ranked components at explaining y.
#include <cmath>
#include <cstdio>
#include <vector>
#include <random>
// Power iteration for the top eigenvector of a symmetric p x p matrix (X^T X here) --
// a from-scratch substitute for a full SVD when only the top direction is needed.
std::vector<double> topEigenvector(const std::vector<std::vector<double>>& A, int iters = 200) {
int p = static_cast<int>(A.size());
std::vector<double> v(p, 1.0 / std::sqrt(static_cast<double>(p)));
for (int it = 0; it < iters; ++it) {
std::vector<double> Av(p, 0.0);
for (int i = 0; i < p; ++i)
for (int j = 0; j < p; ++j) Av[i] += A[i][j] * v[j];
double norm = 0.0;
for (double x : Av) norm += x * x;
norm = std::sqrt(norm);
for (int i = 0; i < p; ++i) v[i] = Av[i] / norm;
}
return v;
}
int main() {
std::mt19937 rng(0);
std::normal_distribution<double> backgroundDist(0.0, 3.0);
std::normal_distribution<double> relevantDist(0.0, 0.4);
std::normal_distribution<double> smallNoise(0.0, 0.2);
const int n = 200, p = 4;
std::vector<std::vector<double>> X(n, std::vector<double>(p));
std::vector<double> y(n);
for (int i = 0; i < n; ++i) {
double background = backgroundDist(rng);
double relevant = relevantDist(rng);
X[i][0] = background + smallNoise(rng);
X[i][1] = background * 0.9 + smallNoise(rng);
X[i][2] = relevant + smallNoise(rng);
X[i][3] = relevant * 0.8 + smallNoise(rng);
y[i] = 5.0 * relevant + smallNoise(rng);
}
std::vector<double> colMean(p, 0.0);
double yMean = 0.0;
for (int i = 0; i < n; ++i) {
for (int j = 0; j < p; ++j) colMean[j] += X[i][j];
yMean += y[i];
}
for (int j = 0; j < p; ++j) colMean[j] /= n;
yMean /= n;
for (int i = 0; i < n; ++i) {
for (int j = 0; j < p; ++j) X[i][j] -= colMean[j];
y[i] -= yMean;
}
// ---- PCA direction: top eigenvector of X^T X, via power iteration ----
std::vector<std::vector<double>> XtX(p, std::vector<double>(p, 0.0));
for (int i = 0; i < n; ++i)
for (int a = 0; a < p; ++a)
for (int b = 0; b < p; ++b) XtX[a][b] += X[i][a] * X[i][b];
std::vector<double> wPca = topEigenvector(XtX);
// ---- PLS direction: the direct closed form, w1 = X^T y / ||X^T y|| ----
std::vector<double> wPls(p, 0.0);
for (int i = 0; i < n; ++i)
for (int j = 0; j < p; ++j) wPls[j] += X[i][j] * y[i];
double wPlsNorm = 0.0;
for (double v : wPls) wPlsNorm += v * v;
wPlsNorm = std::sqrt(wPlsNorm);
for (double& v : wPls) v /= wPlsNorm;
auto fitAndScore = [&](const std::vector<double>& w) {
std::vector<double> z(n);
for (int i = 0; i < n; ++i) {
double dot = 0.0;
for (int j = 0; j < p; ++j) dot += X[i][j] * w[j];
z[i] = dot;
}
double zz = 0.0, zy = 0.0;
for (int i = 0; i < n; ++i) { zz += z[i] * z[i]; zy += z[i] * y[i]; }
double beta = zy / zz;
double sse = 0.0;
for (int i = 0; i < n; ++i) {
double resid = y[i] - beta * z[i];
sse += resid * resid;
}
return sse;
};
double sst = 0.0;
for (double v : y) sst += v * v;
double ssePca = fitAndScore(wPca);
double ssePls = fitAndScore(wPls);
std::printf("PCR, 1 variance-ranked component: R^2 = %.3f\n", 1.0 - ssePca / sst);
std::printf("PLS, 1 covariance-seeking component: R^2 = %.3f\n", 1.0 - ssePls / sst);
return 0;
}- Chemometrics and spectroscopy — PLS's original and still dominant home. A near-infrared spectrum gives thousands of highly correlated wavelength-intensity readings per sample, often for only a few dozen samples, and the goal is predicting a chemical concentration (sugar content, protein level, drug purity). This is the textbook
p \gg nsetting PLS was invented for. - Genomics — regressing a phenotype or outcome on thousands of correlated gene-expression measurements, where most genes co-vary in large correlated blocks and only a handful of underlying biological pathways actually drive the outcome.
- Any p \gg n regression setting generally — more features than samples is exactly where ridge and lasso (earlier in this module) are the standard alternative toolkit; PCR/PLS and ridge/lasso are two different, non-exclusive families of answers to the same underlying instability problem, and it's common practice to try both and compare via cross-validation.
- Econometric forecasting from dozens or hundreds of correlated macroeconomic indicators (interest rates, price indices, employment measures) — PCR/PLS factor models are a standard way to forecast a single target series from a large, collinear panel of predictors.
- Industrial process monitoring and multivariate calibration — sensor arrays on a manufacturing line often produce dozens of correlated readings, and PLS calibration models predict a quality metric (yield, purity, defect rate) from that whole correlated sensor bank at once.
- Marketing mix modeling — many advertising-spend channels that move together (a company-wide budget cycle correlates spend across TV, search, and social all at once), with the actual goal of isolating which combination of channels drives sales.
- Choosing PCR's number of components
kby how much ofX's variance is explained (e.g. "the top 3 components explain 95% of the variance, sok=3") instead of by cross-validated predictive performance ony. These are not the same criterion — that's exactly the derivation's point: a direction with almost none ofX's variance can still be the one direction that actually predictsy, and "percent of X-variance explained" has no way to notice that. - Assuming PLS is unconditionally superior to PCR because "it uses
y." Usingyto choose directions is precisely what makes PLS more prone to overfitting on small samples — every extra look atyduring direction-selection is another chance to fit noise in the training set rather than real signal, a genuine trade-off, not a strict win. With very smalln, PCR's y-blind directions can sometimes generalize better for exactly this reason. - Forgetting to standardize features (mean-center, and usually scale to unit variance) before computing PCA or PLS directions. Both methods are scale-sensitive: a feature measured in different units, or simply larger in raw magnitude, will dominate the variance or covariance calculation for reasons that have nothing to do with its actual relevance — the same standardization pitfall that applies to ridge and lasso.
Going deeper
Push PLS's number of components k all the way up to the full rank of X, and PLS stops discarding anything at all — it reproduces the plain OLS fit exactly. That gives PLS a genuine structural parallel to ridge and lasso's own regularization-strength spectrum, developed earlier in this module: at k=1 PLS is maximally compressed (maximum shrinkage, one direction only), and as k grows toward full rank it interpolates smoothly toward the completely unregularized OLS answer — a full spectrum from heavy shrinkage to none at all, exactly like sweeping ridge's λ from large to zero. The mechanism is just different: ridge and lasso regularize through a penalty term added to the loss, while PCR and PLS regularize through dimensionality — restricting the model to a subspace, then widening that subspace toward the full feature space. Same spectrum of behavior, reached by a structurally different route.
A dataset has 500 highly correlated features and only 40 samples. Your colleague fits PCR, picks k so that the top components explain 99% of the X-variance, and reports a great training fit. What's the likely problem, and how would PLS's approach to choosing directions differ?
The 99%-of-X-variance criterion says nothing about y at all — it's entirely possible (and, with 500 correlated features and only 40 samples, quite likely) that the handful of directions actually predictive of y carry very little of X's total variance and were never included, while the components that were included are dominated by structure in X that has nothing to do with y. A great training fit under this setup is also suspect on its own: with only 40 samples, cross-validated performance on y (not X-variance explained) is the criterion that should have driven the choice of k. PLS chooses its first direction as the (normalized) vector of feature-target covariances, X^Ty, directly — it looks at y from step one rather than deciding on directions from X alone and hoping they happen to matter, though PLS has its own small-sample risk: with only 40 points, its y-aware directions have more freedom to fit noise, so cross-validation matters for PLS too, just for a different reason.
PCR and PLS both replace a pile of raw, correlated features with a small number of derived latent directions and regress on those — the same instinct behind ridge and lasso, reached by compressing dimensionality instead of penalizing coefficients. The one design choice that separates them is what "best direction" means: PCR asks purely "where does X vary the most?", computed via PCA (Linear Algebra for ML, 1.10) with no reference to y at all; PLS asks "where does X covary the most with y?", and its very first direction — derived above as exactly X^Ty, normalized — is built from y from the first step. That's why PLS usually needs fewer components, why PCR's directions stay interpretable as "directions of maximum variance" while PLS's don't, and why neither one is unconditionally the right default — the choice between them, and the choice of how many components to keep, both belong to cross-validation on y, not to any property of X examined on its own.