Trace of a Matrix
The sum of the diagonal — and why it shows up in every regularization derivation.
On this page
Beginner: the trace of a square matrix is simply the sum of its diagonal entries. That's the entire definition — every off-diagonal entry is completely ignored.
Intermediate: despite being so simple to compute, the trace has a surprisingly useful property: trace(AB) = trace(BA), even though AB ≠ BA in general (section 1.5). This "cyclic property" extends to any number of matrices multiplied in a chain, as long as you only rotate the order rather than reversing it — and it's what makes the trace so convenient inside derivations.
Advanced: the trace also equals the sum of a matrix's eigenvalues (just as the determinant, section 1.12, equals their product) — a fact that holds regardless of whether the matrix is diagonalizable, and connects this simple sum directly back to the deep structure of the matrix.
Let A be m×n and B be n×m, so both AB (m×m) and BA (n×n) are square and have well-defined traces. Write out the diagonal entry of AB at row i, then sum over i:
The entire proof is just swapping the order of a double sum — nothing about matrix multiplication's non-commutativity is violated, because trace(AB) and trace(BA) are scalars built from the exact same set of products A_{ij}B_{ji}, just added up in a different grouping. The general cyclic property trace(ABC) = trace(BCA) = trace(CAB) follows by treating BC as a single matrix and applying this two-matrix result once.
Where this is used: this exact swap-the-sum trick is the standard way to derive gradients of scalar loss functions with respect to matrices (section 1.17) — rewriting xᵀAx as trace(Axxᵀ) converts an expression that's awkward to differentiate directly into one with a known matrix-derivative identity.
Regenerate the matrix — the trace only ever sums the highlighted cells, no matter what the rest contains.
These aren't coincidences to memorize — they're the same underlying algebraic fact showing up in three different-looking formulas.
import numpy as np
A = np.random.rand(3, 3)
B = np.random.rand(3, 3)
print(np.trace(A @ B), np.trace(B @ A)) # equal, even though A@B != B@A
eigvals = np.linalg.eigvals(A)
print(np.isclose(np.trace(A), eigvals.sum().real)) # True
# Frobenius norm (section 1.20) via trace:
frob_via_trace = np.sqrt(np.trace(A.T @ A))
print(np.isclose(frob_via_trace, np.linalg.norm(A, 'fro'))) # True- The KL divergence between two multivariate Gaussians — a formula that appears constantly in variational autoencoders and Bayesian ML — includes a
trace(Σ₂⁻¹Σ₁)term directly. - Weight decay / L2 regularization of a weight matrix is literally
trace(WᵀW), connecting straight back to the Frobenius norm of section 1.20. - The cyclic property is the standard trick used to simplify matrix-calculus derivations (section 1.17) that would otherwise involve unwieldy chains of matrix products.
- Assuming
trace(ABC) = trace(CBA)— the cyclic property only permits rotations of the order (ABC → BCA → CAB), not arbitrary reordering or reversal. - Forgetting the trace is only defined for square matrices.
Going deeper
The cyclic property of trace is exactly why trace(xᵀAx) can be rewritten as trace(Axxᵀ) — a rearrangement used constantly to convert an awkward scalar expression into a matrix-calculus-friendly form during gradient derivations.
At the master level: the trace of a matrix is invariant under a change of basis (section 1.2's expert note) — trace(P⁻¹AP) = trace(A) for any invertible P — exactly like the determinant, and for the same underlying reason: both are functions purely of the eigenvalues, which don't depend on which coordinate system you chose to describe the matrix in.