Vector Spaces, Span, Basis & Rank
What it means for features to be genuinely independent.
On this page
Beginner: a vector space is basically "everywhere you could possibly reach" by adding and scaling a set of vectors. The span of a set of vectors is every combination you can build from them. A basis is the smallest, most efficient set of vectors needed to reach everywhere in that space — no redundancy. The rank of a matrix tells you how many truly independent directions it actually covers, even if it has far more rows or columns than that.
Intermediate: a useful analogy: imagine trying to describe every point on a table using only "steps north" and "steps east." Those two directions form a basis for the table's 2D surface — you can reach anywhere with just those two. Adding a third instruction, "steps northeast," doesn't let you reach anywhere new — it's redundant, since northeast is just some combination of north and east.
Advanced: a set of vectors is only a valid basis if none of them are redundant like that (formally, "linearly independent") and together they span the whole space. Every basis of a given space has exactly the same number of vectors — that number is the space's dimension, and it's a genuine invariant, independent of which specific basis you choose to describe it with.
Rank = the number of linearly independent rows/columns = the true dimensionality of the information a matrix holds. For an m×n matrix, rank is always at most min(m, n) — you can never have more independent directions than you have rows or columns to define them.
For an m×n matrix A, define the null space as every input vector A sends to zero ({x : Ax = 0}), and the nullity as that space's dimension. The theorem states:
Why: pick a basis for the null space (nullity vectors), then extend it to a full basis of the entire n-dimensional input space by adding more vectors (always possible — any independent set can be extended to a basis). Applying A to this extended basis: the null-space vectors all map to 0, contributing nothing new, while the remaining vectors map to a set that turns out to be exactly a basis for A's output (column) space — a short argument shows they must be independent, since any dependency among their images would pull a nonzero combination of them back into the null space, contradicting how they were chosen. So the number of "leftover" basis vectors equals rank(A), and nullity + rank = n follows immediately by counting.
Where this is used: this is the precise reason a rank-deficient matrix (section 1.12's determinant-zero case) has a nontrivial null space — it directly explains why Ax = b has infinitely many solutions (not zero, not exactly one) whenever A is singular: the null space's dimension tells you exactly how many free directions of ambiguity exist.
Vectors [1,2] and [2,4] look like two different directions, but [2,4] = 2×[1,2] — they point the exact same way. Their span is just a single line, not a plane, and the matrix [[1,2],[2,4]] has rank 1, not 2.
Contrast with [1,2] and [3,1]: no scalar multiple of one gives the other, so together they can reach every point in the 2D plane — this matrix has rank 2 (full rank for a 2×2 matrix).
Drag either vector around the circle — the rank is computed live from the actual determinant, not just eyeballed.
This exact check — matrix_rank on a feature matrix — is a fast way to catch multicollinearity before it silently breaks a linear regression.
import numpy as np
redundant = np.array([[1, 2], [2, 4]])
independent = np.array([[1, 2], [3, 1]])
print(np.linalg.matrix_rank(redundant)) # 1
print(np.linalg.matrix_rank(independent)) # 2
# A quick real-world check: are two features collinear?
celsius = np.array([0, 10, 20, 30])
fahrenheit = celsius * 9/5 + 32
X = np.column_stack([celsius, fahrenheit])
print(np.linalg.matrix_rank(X)) # 1 -> pure duplicate information- If two features are "temperature in °C" and "temperature in °F," they carry identical information (one is a linear function of the other) — together they only span a 1-D line, not a 2-D plane. This is multicollinearity, and it's exactly why
Xᵀ Xcan become singular (impossible to invert) in the normal equation from section 1.6, which is what motivates ridge regression's small diagonal correction. - Feature engineering pipelines routinely drop one-hot-encoded columns (e.g. keeping only n−1 of n categories) specifically to avoid rank-deficiency in the resulting design matrix.
- Dimensionality reduction (PCA, autoencoders) is fundamentally about finding a lower-rank approximation of your data that still captures most of its span.
- Assuming "more features" always means "more information" — redundant features add zero rank and zero new information, only noise and computational cost.
- Not checking for near-collinearity (rank technically full, but barely) — this still causes numerical instability even when
matrix_rankdoesn't flag it outright.
Going deeper
Rank directly determines whether Xᵀ X is invertible. Rank-deficient (collinear) features make it singular — regularization adds a small value to the diagonal specifically to make it invertible again, a trick sometimes called a "ridge."
In deep learning, "low-rank adaptation" (LoRA) exploits exactly this idea for efficient fine-tuning: instead of updating a full, huge weight matrix, you learn two small low-rank matrices whose product approximates the needed update — dramatically cutting the number of trainable parameters while barely affecting quality.
At the master level: rank is only well-defined exactly in infinite-precision arithmetic — in floating point, "numerical rank" is determined by counting singular values (section 1.10) above some small tolerance, since real-world data almost never produces an exactly-zero singular value, only a very small one. This is the practical bridge between the clean textbook definition of rank and how it's actually computed in every numerical library.