Singular Value Decomposition (SVD)
Any matrix, broken into rotate → stretch → rotate.
On this page
Beginner: every matrix — square or not, even if it's rank-deficient — can be broken into three simple pieces: a rotation, a stretch along perpendicular axes, and another rotation. Singular Value Decomposition (SVD) finds exactly those three pieces. It's like eigenvectors (section 1.7) but works for any matrix, not just square ones.
Intermediate: why does this matter so much? Because it means literally any linear transformation — no matter how complicated it looks — is secretly just three simple, well-understood steps in disguise. Once you have a matrix's SVD, questions that seem hard ("what's the best rank-5 approximation of this?", "how sensitive is this system to small input errors?") all become nearly trivial to answer by reading off numbers from U, Σ, and V.
Advanced: the Eckart-Young theorem proves something remarkable — truncating an SVD to its top k singular values gives, provably, the best possible rank-k approximation of the original matrix, in a precise mathematical sense (minimizing reconstruction error). This isn't a heuristic; it's an exact optimality guarantee, which is why SVD-based compression is a genuine gold standard rather than just "one reasonable option."
U and V are rotation (orthogonal) matrices; Σ is a diagonal matrix of "singular values" — the stretch amount along each axis, largest to smallest. Unlike eigen-decomposition, SVD always exists for every matrix of every shape, with no exceptions and no need for the matrix to be square or diagonalizable.
Start from AᵀA — always defined, for any shape of A. It's symmetric ((AᵀA)ᵀ = AᵀA) and positive semi-definite (section 1.15), since for any x, xᵀ(AᵀA)x = (Ax)ᵀ(Ax) = ‖Ax‖² ≥ 0. By the spectral theorem (section 1.7's expert note), a symmetric matrix always has a full set of orthonormal eigenvectors v₁,…,vₙ with real, non-negative eigenvalues λ₁,…,λₙ.
Define singular values σᵢ = √λᵢ, and for each σᵢ > 0, define uᵢ = Avᵢ/σᵢ. Check these uᵢ are themselves orthonormal:
which is 0 for i≠j (eigenvectors of a symmetric matrix are orthogonal) and exactly 1 when i=j (since λᵢ = σᵢ²). With U's columns as these uᵢ, V's columns as the vᵢ, and Σ holding the σᵢ, the construction directly gives A = UΣVᵀ — and since AᵀA exists and is always diagonalizable for any matrix A of any shape, this construction never fails.
Where this is used: this is exactly the computation running inside np.linalg.svd conceptually (real implementations use more numerically stable algorithms, but this is the mathematical guarantee behind them) — and it's why SVD, unlike plain eigen-decomposition (section 1.7), never needs to worry about non-square or non-diagonalizable inputs.
Keeping only the largest singular value of a matrix (and dropping the rest) gives the best possible rank-1 approximation of that matrix — the closest simpler version of it you can build. Keep the top few singular values instead of all of them, and you get a compressed approximation that still captures most of the original structure.
Concretely, if an image's singular values are [420, 85, 12, 3, 0.4, ...], the first one or two already dominate — reconstructing the image from just the top 20 out of, say, 500 singular values often looks nearly identical to the original, because the rest contribute almost no visual information.
Drag the slider yourself — a circle of points gets rotated (Vᵀ), stretched into an ellipse (Σ), then rotated again (U). That sequence IS the SVD.
This is a real, working image/data compressor in six lines — the same idea scales up to production recommender systems and topic models, just with far larger matrices.
import numpy as np
# Any matrix, e.g. a 100x100 "image"
A = np.random.rand(100, 100)
U, S, Vt = np.linalg.svd(A)
k = 10 # keep only the top 10 singular values
A_approx = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]
original_size = A.size # 10,000 numbers
compressed_size = U[:, :k].size + k + Vt[:k, :].size
print(f"{compressed_size} numbers instead of {original_size}")
print("Reconstruction error:", np.linalg.norm(A - A_approx))- Recommender systems (Netflix-style) factor a giant, sparse user-movie matrix via SVD to discover hidden "taste dimensions" (e.g. "prefers action movies") without anyone labeling them — at real-world scale, this is done via randomized SVD (section 1.30), not the exact version shown here.
- Image compression keeps only the top singular values of a pixel matrix to reconstruct a close approximation using a fraction of the data. If you want the factors themselves to stay non-negative and interpretable rather than allowing negative entries, that's non-negative matrix factorization instead (section 1.31).
- Latent Semantic Analysis in NLP factors word-document matrices the same way to find topics, entirely unsupervised.
- Noise reduction — since noise tends to spread evenly across all singular values while signal concentrates in the largest ones, truncating small singular values is a simple, effective denoising technique.
- Embedding alignment — the SVD of a cross-covariance matrix also gives the exact, closed-form solution to the orthogonal Procrustes problem (section 1.33) of rotating one point set onto another.
- Forgetting that
np.linalg.svdreturnsVᵀalready transposed, notV— a very common source of off-by-transpose bugs. - Choosing k (how many singular values to keep) arbitrarily instead of by inspecting how quickly the singular values decay — plot them; the "elbow" tells you where information genuinely runs out.
Going deeper
PCA is literally SVD applied to the mean-centered data matrix — the right singular vectors V are the principal components. Libraries typically compute PCA via SVD rather than eigen-decomposing the covariance matrix directly, for better numerical stability.
The Moore-Penrose pseudo-inverse — the closest thing to "dividing" by a non-square or singular matrix — is built directly from the SVD: invert the non-zero singular values, transpose the shape, and reassemble. This is what lets you solve least-squares problems even when a matrix has no true inverse at all.
At the master level: the ratio of the largest to smallest singular value is the matrix's condition number — a huge condition number means the matrix is nearly singular and small input errors get massively amplified in any solution computed from it. This single number is the standard, rigorous way numerical analysts quantify "how trustworthy" a computed result really is, and it's computed via SVD in essentially every serious linear algebra library. For large random matrices, the statistical distribution of singular values follows the Marchenko-Pastur law — the rectangular-matrix cousin of the eigenvalue semicircle law covered in section 1.19.