Why It Matters in ML/DL
Tying the whole chapter together.
On this page
Every dataset is a matrix. Every data point is a vector. Every neural network layer is a matrix multiplication plus a bias vector (1.5). Similarity between two things is a dot product (1.3). Compressing data or finding its most important patterns is eigen-decomposition (1.7) or SVD (1.10). Knowing whether your features carry real, independent information — or just redundant copies of each other — is rank (1.9). Keeping a model from overfitting is norm-based regularization (1.8). If you genuinely understand vectors, matrices, dot products, eigenvectors, rank, and SVD, you can read almost any ML paper's math section without getting lost.
It's also worth naming the two big ideas one level up, because they'll keep reappearing: linear algebra is the language of transformation (matrices moving vectors around, which is what every neural network layer does) and of structure (rank, span, and eigenvectors describing what information is actually present, independent of how it's stored). Almost every advanced ML concept — attention, convolution, embeddings, factorization, gradient descent itself — is one of these two ideas wearing a different name.
Every single line of this snippet is a concept from this chapter, applied to the same 200×5 dataset — this is genuinely what a real ML pipeline's early stages look like under the hood.
import numpy as np
X = np.random.rand(200, 5) # 200 samples, 5 features (a matrix)
x0 = X[0] # one data point (a vector, 1.2)
sim = np.dot(x0, X[1]) / (np.linalg.norm(x0) * np.linalg.norm(X[1])) # 1.3
W = np.random.rand(5, 3) # a "layer" (1.4, 1.5)
layer_out = X @ W
Xc = X - X.mean(axis=0) # center the data
cov = Xc.T @ Xc / len(X) # covariance matrix (1.6)
eigvals, eigvecs = np.linalg.eig(cov) # PCA directions (1.7)
reg_penalty = np.linalg.norm(W, ord=2) # weight regularization (1.8)
rank = np.linalg.matrix_rank(X) # true information content (1.9)
U, S, Vt = np.linalg.svd(Xc) # PCA, done the stable way (1.10)Lessons 1.2–1.11 are the foundation almost every intro course stops at. The rest of this chapter covers what's genuinely needed to go from "comfortable" to "expert" — the parts of linear algebra that show up in research papers, production ML systems, and technical interviews, but rarely in beginner material:
- 1.12 Determinants — the precise, computable meaning of "singular."
- 1.13 LU Decomposition — how
Ax = bis actually solved, without ever inverting A. - 1.14 QR & Gram-Schmidt — orthogonalization, and the numerically stable way to do regression.
- 1.15 Positive Definite Matrices & Cholesky — the matrix version of "positive," and how Gaussians are sampled.
- 1.16 Quadratic Forms & Convexity — why some loss landscapes are easy and most deep learning isn't.
- 1.17 Matrix Calculus — the chain rule in matrix form, which is exactly what backpropagation is.
- 1.18 Einsum & Tensor Contractions — the real notation behind attention and convolution.
- 1.19 Random Matrix Theory — why weight initialization is scaled the way it is.
- 1.20 Matrix Norms — Frobenius, spectral, and nuclear norms, and spectral normalization.
- 1.21 Numerical Stability — the log-sum-exp trick behind every stable softmax.
- 1.22 Sparse Matrices — why real-world ML data is almost always mostly zeros.
- 1.23 Trace of a Matrix — the sum of the diagonal, and its surprising reach.
- 1.24 Power Iteration — the simple algorithm behind PageRank and fast approximate PCA.
- 1.25 Kernel Methods — separating data no straight line ever could, without an explicit lift.
- 1.26 Whitening — turning correlated, stretched data into an isotropic cloud.
- 1.27 The Woodbury Identity — updating an inverse cheaply, the trick behind Kalman filters.
- 1.28 Perron-Frobenius Theorem — the guarantee that makes PageRank's steady state well-defined.
- 1.29 Generalized Eigenvalue Problems — what LDA and CCA actually solve.
- 1.30 Randomized Linear Algebra — how PCA/SVD scale to millions of rows.
- 1.31 Non-negative Matrix Factorization — SVD's interpretable cousin.
- 1.32 Hessian-Vector Products — getting Hv without ever forming H.
- 1.33 Orthogonal Procrustes — the exact way to align two shapes or embedding spaces.
- 1.34 Conjugate Gradient — solving Ax = b at massive scale, beating gradient descent's zig-zag.
- 1.35 Triangular Jacobians & Normalizing Flows — why flow-based generative models are built the exact way they are.
- Calculus — gradients are vectors of partial derivatives; backpropagation is the chain rule applied through a graph of matrix multiplications (1.17 is the bridge into this).
- Probability & statistics — covariance matrices, the multivariate normal distribution, and maximum likelihood estimation all lean directly on this chapter (1.15 is the bridge into this).
- Optimization — gradient descent, Newton's method, and convexity all reason about functions using the vectors and matrices introduced here (1.16 and 1.17 are the bridge into this).
Linear algebra isn't a hurdle before ML — it is the notation ML is written in. Every section in this chapter reappears, unmodified, the moment you open a deep learning paper.