Whitening & Decorrelation Transforms
Turning correlated, stretched data into an isotropic unit-variance cloud.
On this page
Beginner: real data is often correlated — some features stretch out more than others, and directions can be tangled together (like the elongated, tilted point clouds seen throughout this chapter). Whitening is a transformation that removes both effects at once: afterward, every direction has equal spread (unit variance), and no two directions are correlated with each other.
Intermediate: the recipe uses the tools from earlier sections directly: eigendecompose (section 1.7) the data's covariance matrix to find its principal axes, then rescale along each axis by 1/√λᵢ — dividing out exactly the amount of spread that axis originally had.
Advanced: there isn't one unique whitening transform — PCA whitening (rotate to the eigenbasis, then scale) and ZCA whitening (do the same, but rotate back to the original coordinate system afterward) both produce decorrelated, unit-variance data, but ZCA keeps the result visually closer to the original data's orientation, which matters when the whitened output needs to still "look like" the input (a common requirement in image preprocessing).
is computed from the covariance matrix's eigendecomposition: Σ = VΛVᵀ gives Σ⁻¹ᐟ² = VΛ⁻¹ᐟ²Vᵀ — a direct, practical use of the material from section 1.7.
Let Σ = VΛVᵀ be the eigendecomposition of the (symmetric, PSD) covariance matrix, and define y = Σ⁻¹ᐟ²(x − μ) = VΛ⁻¹ᐟ²Vᵀ(x − μ). Compute the covariance of the transformed variable y directly from the definition, using that covariance transforms as Cov(Mx) = M·Cov(x)·Mᵀ for any constant matrix M:
Since VᵀV = I (V is orthogonal), every adjacent VᵀV pair collapses, leaving purely diagonal matrices multiplying each other:
The middle diagonal product is literally λᵢ⁻¹ᐟ² · λᵢ · λᵢ⁻¹ᐟ² = 1 for every eigenvalue, so it's exactly the identity matrix, and VIVᵀ = I too. That's the algebraic guarantee, not just a visual impression from the diagram: after this transform, every direction has variance exactly 1 and every pair of directions has covariance exactly 0.
Where this is used: whitening as a preprocessing step before k-means or nearest-neighbor search (both of which implicitly assume all directions are equally scaled), ZCA whitening in classic image-preprocessing pipelines, and as a component inside independent component analysis (ICA), which requires whitened input before it can separate independent signal sources.
This is exactly what StandardScaler + PCA (or a dedicated whitening transform) does to a dataset before feeding it to a distance-sensitive algorithm.
Note the small + 1e-8 — a direct application of the "jitter" trick from section 1.15, needed because a real covariance matrix can have a tiny near-zero eigenvalue that would otherwise blow up the 1/√λ scaling.
import numpy as np
X = np.random.randn(500, 2) @ np.array([[3, 1], [1, 0.5]]) # correlated, stretched data
Xc = X - X.mean(axis=0)
cov = Xc.T @ Xc / len(X)
eigvals, eigvecs = np.linalg.eigh(cov) # eigh: for symmetric matrices, sorted ascending
# PCA whitening: rotate into eigenbasis, then scale by 1/sqrt(eigenvalue)
X_whitened = Xc @ eigvecs @ np.diag(1.0 / np.sqrt(eigvals + 1e-8))
print(np.cov(X_whitened.T)) # ~identity matrix: unit variance, zero correlation- Preprocessing for classical ML algorithms — distance-based methods (k-nearest-neighbors, k-means) and gradient-based optimizers both tend to perform substantially better on whitened, decorrelated input features.
- Batch normalization in deep learning is a lightweight, per-batch approximation of whitening applied to a layer's activations — it only rescales variance per-feature rather than fully decorrelating, as a much cheaper compromise.
- Independent Component Analysis (ICA), used for tasks like separating mixed audio signals, uses whitening as its standard first preprocessing step before the non-Gaussian independence search begins.
- Whitening using the eigenvalues of a covariance matrix computed on your test set — always compute the whitening transform on training data only, then apply it to test data, to avoid leaking test-set statistics.
- Forgetting the small epsilon regularizer when dividing by
√λ— near-zero eigenvalues (common with correlated or low-rank features) otherwise amplify noise enormously.
Going deeper
Whitening is the same eigen-decomposition machinery as PCA (section 1.7, 1.10), just followed by an extra rescaling step — which is why it's often described as "PCA, then normalize each component."
At the master level: over-aggressive whitening on high-dimensional data with limited samples can badly amplify estimation noise in the smallest eigenvalue directions (since dividing by a poorly estimated small number is numerically unstable) — in practice, whitening is often combined with dimensionality reduction (dropping the smallest-eigenvalue directions entirely, section 1.10) rather than applied to every direction indiscriminately.