Matrix Norms: Frobenius, Spectral & Nuclear
The matrix-sized versions of section 1.8 — and the basis of spectral normalization.
On this page
Beginner: section 1.8 measured the "size" of a vector. Matrices need their own notion of size too — for example, "how big is the error between this matrix and its low-rank approximation?" A matrix norm answers exactly that.
Intermediate: the simplest one, the Frobenius norm, just treats the matrix as one long vector of all its entries and takes the ordinary L2 length of that — squares every entry, sums, square-roots. The spectral norm is different in kind: it's the matrix's largest singular value (section 1.10) — the single biggest amount it can stretch any input vector by.
Advanced: the nuclear norm (sum of all singular values) is the matrix analogue of the L1 norm from section 1.8 — and just as L1 regularization on a vector pushes individual entries to exactly zero (automatic feature selection), nuclear-norm regularization on a matrix pushes singular values to exactly zero, which means it pushes the rank down. This is the mathematical basis of low-rank matrix completion.
Frobenius (F), spectral (2, "operator norm"), and nuclear (*) — all three are defined in terms of the matrix's singular values, and all three collapse to the ordinary vector L2 norm when applied to a matrix with only one column.
All three norms are built from the same list of non-negative singular values σ₁ ≥ σ₂ ≥ ⋯ ≥ σᵣ ≥ 0. The spectral norm is just the largest one, σ₁. The Frobenius norm squares and sums all of them:
— true because every term in the sum is non-negative, so dropping all but the largest can only decrease the total. Taking square roots preserves the inequality, giving the first half.
For the second half, square the nuclear norm directly:
— the cross terms σᵢσⱼ are all non-negative (singular values are never negative), so the squared nuclear norm can only be larger than or equal to the squared Frobenius norm. Square-rooting both sides gives ‖A‖_F ≤ ‖A‖*.
Where this is used: this ordering is exactly why nuclear-norm regularization (used for low-rank matrix completion) is always at least as "strong" a penalty as Frobenius-norm weight decay for the same matrix — it never under-penalizes relative to the simpler norm.
For A = [[3, 0], [4, 5]], the Frobenius norm is the easiest to check by hand: √(3² + 0² + 4² + 5²) = √50 ≈ 7.07 — just treat every entry as one long vector. The spectral and nuclear norms both require the singular values (section 1.10) first. Since AᵀA = [[25, 20], [20, 25]] has eigenvalues 45 and 5, A's singular values are √45 ≈ 6.71 and √5 ≈ 2.24 — so the spectral norm is ‖A‖₂ ≈ 6.71 (the larger one) and the nuclear norm is ‖A‖* ≈ 6.71 + 2.24 = 8.95 (their sum) — confirming the ordering ‖A‖₂ ≤ ‖A‖_F ≤ ‖A‖* from the expert note below (6.71 ≤ 7.07 ≤ 8.95).
np.linalg.norm handles all three with just a different ord argument — worth knowing since the default (no ord at all) silently computes Frobenius for a matrix, not spectral.
import numpy as np
A = np.array([[3., 0.], [4., 5.]])
frob = np.linalg.norm(A, 'fro') # sqrt(9+0+16+25) = sqrt(50)
spectral = np.linalg.norm(A, 2) # largest singular value
nuclear = np.linalg.norm(A, 'nuc') # sum of all singular values
print(frob, spectral, nuclear)
# Spectral norm connects directly to the SVD from section 1.10:
_, S, _ = np.linalg.svd(A)
print(np.isclose(spectral, S.max())) # True- Spectral normalization — a well-known GAN training technique divides each layer's weight matrix by its spectral norm, which caps how much the layer can stretch any input by exactly 1×, directly enforcing a Lipschitz constraint that stabilizes training.
- Matrix completion (the "fill in the missing Netflix ratings" problem) minimizes nuclear norm as a convex proxy for minimizing rank directly, which is otherwise a computationally intractable (NP-hard) objective.
- Weight decay is, technically, Frobenius-norm regularization of a weight matrix — the direct matrix generalization of the L2 vector regularization from section 1.8.
- Assuming "matrix norm" always means Frobenius — many papers and libraries default to spectral norm instead when discussing Lipschitz constraints; always check which is meant.
- Computing the spectral norm as the maximum absolute entry of the matrix — that's not a correct matrix norm at all (fails the sub-multiplicative requirement); it must come from the SVD.
Going deeper
The three norms sandwich each other in a precise way: ‖A‖₂ ≤ ‖A‖_F ≤ ‖A‖* — the spectral norm is always the smallest of the three, the nuclear norm the largest, for any matrix.
At the master level: the Frobenius norm equals √trace(AᵀA) (section 1.23) — connecting matrix norms, the trace, and the SVD into one identity, and explaining why "Frobenius norm of the reconstruction error" and "sum of squared residuals" are the exact same quantity used throughout PCA and matrix factorization literature.