Transpose, Identity & Inverse
Flipping, "doing nothing," and "undoing" a matrix.
On this page
Beginner: the transpose flips a matrix over its diagonal — rows become columns. The identity matrix is the matrix version of the number 1: multiplying anything by it changes nothing. The inverse undoes whatever the original matrix did, the way multiplying by ⅕ undoes multiplying by 5.
Intermediate: these three ideas travel together constantly in practice. You'll see Aᵀ used to reshape data for a multiplication to be valid (fixing a shape mismatch), I used as a "do nothing" placeholder or as a starting point for building other matrices (like a rotation matrix at angle 0), and A⁻¹ used whenever you need to "solve for x" rather than just "compute the output" — the difference between prediction and inference.
Advanced: not every matrix has an inverse — only square matrices can, and even among those, only ones whose rows/columns are all independent (section 1.9). When an inverse doesn't exist, you can't uniquely "undo" the transformation because information was destroyed along the way (multiple different inputs got mapped to the same output). This is precisely the geometric meaning of a matrix being "singular."
Only square matrices can have an inverse — and even then, not all of them do (a "singular" matrix has none, typically because its rows or columns are redundant — see section 1.9 on rank). A matrix's transpose always exists, for any shape: transposing an m×n matrix gives an n×m matrix.
Claim: if A and B are both invertible, then AB is invertible and its inverse is B⁻¹A⁻¹ — note the order flips, exactly like undoing "put on socks, then shoes" by first removing shoes, then socks.
Proof: to show B⁻¹A⁻¹ really is the inverse of AB, it's enough to check it satisfies the defining property directly — multiply them together and confirm the result is the identity:
using associativity (proved in section 1.5) to regroup the middle product first. The same check in the other order, (B⁻¹A⁻¹)(AB) = B⁻¹(A⁻¹A)B = B⁻¹IB = B⁻¹B = I, confirms it from both sides, which is exactly what "is the inverse" requires.
Where this is used: this rule is why, when undoing a sequence of transformations (a rotation followed by a scale, several neural network layers, a chain of coordinate changes), you must invert each step and also reverse their order — never just invert each piece independently and multiply in the original order.
Rows become columns — cell (r, c) moves to position (c, r). Edit the matrix, then replay.
NumPy raises LinAlgError for singular matrices rather than silently returning a wrong answer — a good habit is to wrap real production code in exactly this kind of try/except.
import numpy as np
A = np.array([[4.0, 7.0], [2.0, 6.0]])
print(A.T) # transpose: rows <-> columns
print(np.linalg.inv(A)) # inverse: A @ inv(A) == I
print(A @ np.linalg.inv(A)) # -> [[1, 0], [0, 1]] (up to float rounding)
try:
singular = np.array([[1, 2], [2, 4]]) # rank 1 -> no inverse exists
np.linalg.inv(singular)
except np.linalg.LinAlgError as e:
print("Singular matrix:", e)Linear regression's normal equation uses exactly these ideas:
This directly solves for the best-fit weights w from your data matrix X and targets y — no trial and error needed for small datasets. Every piece has a role: Xᵀ reshapes the problem so the multiplication is valid, (XᵀX)⁻¹ "undoes" the data's own structure, and the whole expression is really just algebraically solving Xw = y for w.
- Covariance matrices are always computed via a transpose:
Cov = (1/n) XᵀXon centered data. - Graphics engines use the transpose of a rotation matrix as a cheap way to compute its inverse (rotation matrices are "orthogonal," so transpose = inverse for them — no expensive inversion needed).
- Calling
np.linalg.inv()on a large matrix in production code — it's slow and numerically unstable at scale; usenp.linalg.solve(A, b)to solveAx = bdirectly instead of computing a full inverse. - Assuming every square matrix is invertible — always check for singularity (or rank deficiency, section 1.9) before relying on an inverse existing.
Going deeper
Real ML libraries rarely compute a literal inverse for large problems — it's numerically unstable and slow. They use decompositions like LU (section 1.13), QR (section 1.14), or SVD (section 1.10) instead, or just solve iteratively with gradient descent.
A useful mental shortcut: for an orthogonal matrix (one whose rows/columns are unit vectors, all perpendicular to each other — like a pure rotation), the inverse is always just the transpose: A⁻¹ = Aᵀ. This is one of the main reasons orthogonal matrices are so beloved in numerical computing — inversion, normally expensive, becomes free.
At the master level: when a true inverse doesn't exist (non-square or singular matrices), the Moore-Penrose pseudo-inverse (np.linalg.pinv) generalizes the concept, built from the SVD (section 1.10) — it's what "least squares" solutions actually rely on under the hood in every serious numerical library.