Trace of a Matrix
The sum of the diagonal — and why it shows up in every regularization derivation.
On this page
Beginner: the trace of a square matrix is simply the sum of its diagonal entries. That's the entire definition — every off-diagonal entry is completely ignored.
Intermediate: despite being so simple to compute, the trace has a surprisingly useful property: trace(AB) = trace(BA), even though AB ≠ BA in general (section 1.5). This "cyclic property" extends to any number of matrices multiplied in a chain, as long as you only rotate the order rather than reversing it — and it's what makes the trace so convenient inside derivations.
Advanced: the trace also equals the sum of a matrix's eigenvalues (just as the determinant, section 1.12, equals their product) — a fact that holds regardless of whether the matrix is diagonalizable, and connects this simple sum directly back to the deep structure of the matrix.
Let A be m×n and B be n×m, so both AB (m×m) and BA (n×n) are square and have well-defined traces. Write out the diagonal entry of AB at row i, then sum over i:
The entire proof is just swapping the order of a double sum — nothing about matrix multiplication's non-commutativity is violated, because trace(AB) and trace(BA) are scalars built from the exact same set of products A_{ij}B_{ji}, just added up in a different grouping. The general cyclic property trace(ABC) = trace(BCA) = trace(CAB) follows by treating BC as a single matrix and applying this two-matrix result once.
Where this is used: this exact swap-the-sum trick is the standard way to derive gradients of scalar loss functions with respect to matrices (section 1.17) — rewriting xᵀAx as trace(Axxᵀ) converts an expression that's awkward to differentiate directly into one with a known matrix-derivative identity.
Regenerate the matrix — the trace only ever sums the highlighted cells, no matter what the rest contains.
These aren't coincidences to memorize — they're the same underlying algebraic fact showing up in three different-looking formulas.
#include <cmath>
#include <cstdio>
#include <vector>
#include <random>
using Matrix = std::vector<std::vector<double>>;
Matrix randMatrix(int n, std::mt19937& rng) {
std::uniform_real_distribution<double> unit(0.0, 1.0);
Matrix A(n, std::vector<double>(n));
for (auto& row : A) for (double& v : row) v = unit(rng);
return A;
}
Matrix matmul(const Matrix& A, const Matrix& B) {
int n = A.size();
Matrix C(n, std::vector<double>(n, 0.0));
for (int i = 0; i < n; ++i)
for (int k = 0; k < n; ++k)
for (int j = 0; j < n; ++j)
C[i][j] += A[i][k] * B[k][j];
return C;
}
Matrix transpose(const Matrix& A) {
int n = A.size();
Matrix T(n, std::vector<double>(n));
for (int i = 0; i < n; ++i)
for (int j = 0; j < n; ++j) T[j][i] = A[i][j];
return T;
}
double trace(const Matrix& A) {
double s = 0.0;
for (size_t i = 0; i < A.size(); ++i) s += A[i][i];
return s;
}
int main() {
std::mt19937 rng(0);
Matrix A = randMatrix(3, rng);
Matrix B = randMatrix(3, rng);
std::printf("trace(AB)=%.6f trace(BA)=%.6f\n", trace(matmul(A, B)), trace(matmul(B, A)));
Matrix AtA = matmul(transpose(A), A);
double frobViaTrace = std::sqrt(trace(AtA));
double frobDirect = 0.0;
for (const auto& row : A) for (double v : row) frobDirect += v * v;
frobDirect = std::sqrt(frobDirect);
std::printf("frob_via_trace=%.6f frob_direct=%.6f\n", frobViaTrace, frobDirect);
return 0;
}- The KL divergence between two multivariate Gaussians — a formula that appears constantly in variational autoencoders and Bayesian ML — includes a
trace(Σ₂⁻¹Σ₁)term directly. - Weight decay / L2 regularization of a weight matrix is literally
trace(WᵀW), connecting straight back to the Frobenius norm of section 1.20. - The cyclic property is the standard trick used to simplify matrix-calculus derivations (section 1.17) that would otherwise involve unwieldy chains of matrix products.
- Assuming
trace(ABC) = trace(CBA)— the cyclic property only permits rotations of the order (ABC → BCA → CAB), not arbitrary reordering or reversal. - Forgetting the trace is only defined for square matrices.
Going deeper
The cyclic property of trace is exactly why trace(xᵀAx) can be rewritten as trace(Axxᵀ) — a rearrangement used constantly to convert an awkward scalar expression into a matrix-calculus-friendly form during gradient derivations.
At the master level: the trace of a matrix is invariant under a change of basis (section 1.2's expert note) — trace(P⁻¹AP) = trace(A) for any invertible P — exactly like the determinant, and for the same underlying reason: both are functions purely of the eigenvalues, which don't depend on which coordinate system you chose to describe the matrix in.