Matrix Operations
How matrices combine — and how neural networks actually compute.
On this page
Beginner: matrix addition is straightforward — add matching cells, and it only works when both matrices are exactly the same shape. Matrix multiplication is the one to understand deeply: to get one output number, take a full row of the first matrix, a full column of the second, and compute their dot product.
Intermediate: the shape rule is worth memorizing precisely, because it's the single most common source of bugs when building neural networks by hand: multiplying an (m×n) matrix by an (n×p) matrix requires the inner dimensions to match (both n), and the result is an (m×p) matrix — the outer dimensions "survive." If the inner dimensions don't match, the multiplication is simply undefined, not approximately correct.
Advanced: matrix multiplication composes transformations. If matrix B rotates space and matrix A then stretches it, the single matrix AB does both at once, in that order, to any vector you feed it. This is exactly how a multi-layer neural network's effective transformation could in principle be written as one giant matrix — if there were no non-linear activation functions breaking up the chain (which is precisely why those non-linearities are essential: without them, an arbitrarily deep network would collapse to a single linear transformation).
In plain words: entry (i, j) of the result is "row i of A, dotted with column j of B." Do this for every combination of row and column and you have the full product.
Write both sides in index notation and expand using the definition above. Entry (i, l) of (AB)C:
Multiply through and swap the (finite) order of summation — always valid for finite sums:
Since this holds for every entry (i, l), the two matrices are identical: (AB)C = A(BC).
Where this is used: associativity is why a deep network's layers can be grouped and composed in any order without changing the result, and why libraries can freely choose the cheapest evaluation order for a long chain of matrix products (exactly the contraction-ordering question raised in section 1.18's expert note).
Edit the numbers below and watch the result recompute instantly:
Edit A or v directly — the highlighted-row/column intuition from 1.4 is exactly what's happening under the hood.
This four-line snippet is a full forward pass through one neural network layer — W @ x + b is matrix multiplication plus a bias vector, exactly as described above, and np.maximum(0, z) is the ReLU non-linearity that keeps stacked layers from collapsing into a single linear transformation.
import numpy as np
W = np.array([[0.2, -0.5], [0.8, 0.1]]) # weights, shape (2, 2)
x = np.array([1.0, 2.0]) # input, shape (2,)
b = np.array([0.1, -0.2]) # bias, shape (2,)
z = W @ x + b # matrix multiply then add bias -> shape (2,)
output = np.maximum(0, z) # ReLU activation
print(output)- Every neural network layer computes
output = W·x + b— stack enough of these (with a non-linearity between) and you get a deep network. - GPUs matter specifically because of this operation — they're built with thousands of cores designed to run matrix multiplications in parallel, which is why "more GPU" almost always means "train bigger models faster."
- 3D graphics and game engines multiply every vertex of a 3D model by a 4×4 transformation matrix (combining rotation, scaling, and translation) on every single frame.
- Markov chains use matrix multiplication to advance probability distributions one step in time — multiplying a state vector by a transition matrix repeatedly is literally how PageRank (early Google search ranking) was computed.
- Trying to multiply two matrices whose inner dimensions don't match — always sanity-check shapes first: (m×n)·(n×p) → (m×p).
- Assuming
AB = BA— matrix multiplication is not commutative in general; order matters and changes the result (or breaks the shapes entirely). - Confusing elementwise multiplication (
A * Bin NumPy) with true matrix multiplication (A @ B) — they require different shapes and mean entirely different things.
Going deeper
Matrix multiplication is not commutative — generally AB ≠ BA. Frameworks use batched matrix multiplication (torch.bmm) to apply the same operation across many samples at once.
Naive matrix multiplication is O(n³) for two n×n matrices, but this is a genuinely open area of research — Strassen's algorithm (1969) does better than the naive approach, and in 2022 DeepMind's AlphaTensor discovered even faster multiplication algorithms for specific matrix sizes using reinforcement learning, which is a nice example of ML being used to improve the very linear algebra that powers ML.
At the master/production level: modern NVIDIA GPUs (Volta architecture onward) include Tensor Cores — hardware units that do nothing but general matrix multiply (GEMM), computing an entire small matrix-multiply-accumulate per clock cycle instead of one scalar per core. Mixed-precision training (FP16 or BF16 inputs accumulated in FP32) exists specifically to feed these units efficiently — this hardware detail, one level below any framework code, is a large part of why modern deep learning training is fast enough to be practical at all.