Vector Norms
Measuring the "size" of a vector — and why it controls overfitting.
On this page
Beginner: a norm measures how "big" a vector is. The L1 norm adds up absolute values — like walking city blocks, only horizontal/vertical moves. The L2 norm is straight-line distance — a bird flying directly there.
Intermediate: there's a whole family beyond just L1 and L2: the general Lp norm raises each component to the power p, sums them, then takes the p-th root. As p grows toward infinity, the norm cares less and less about the small entries and more and more about the single largest one — the limiting case, the L∞ norm, is simply "the biggest absolute value in the vector."
Advanced: different norms encode genuinely different notions of "size," and choosing the right one is a real modeling decision, not a formality. L2 penalizes large values disproportionately (squaring amplifies them), which spreads a penalty evenly across all weights; L1's flat linear penalty can push individual weights all the way to exactly zero, which is what makes it useful for automatic feature selection rather than just "shrinkage."
The general form is — setting p=1 or p=2 recovers the two formulas above.
Expand the squared L2 norm of a sum using the dot product directly:
The Cauchy-Schwarz inequality states a·b ≤ ‖a‖‖b‖ for any two vectors (a direct consequence of section 1.3's identity, since cos θ ≤ 1 always). Substituting this bound in:
and since norms are non-negative, taking the square root of both sides preserves the inequality: ‖a+b‖ ≤ ‖a‖+‖b‖.
Where this is used: the triangle inequality is one of the three defining axioms a function must satisfy to legally be called a "norm" or "distance metric" at all — it's what guarantees "going directly somewhere is never longer than a detour," the property every distance-based ML algorithm (k-NN, clustering) silently depends on.
For v = [3, -4]:
| Norm | Calculation | Result |
|---|---|---|
| L1 | |3|+|-4| | 7 |
| L2 | √(9+16) | 5 |
| L∞ | max(|3|,|-4|) | 4 |
Drag the endpoint — orange is the L1 (city-block, staircase) path, green is the L2 (straight-line) distance. Notice L2 is always ≤ L1.
This is literally what Ridge and Lasso regression add to their loss functions — a norm of the weight vector, scaled by a tunable strength λ.
import numpy as np
weights = np.array([2.0, -0.1, 3.5, 0.0, -1.2])
l1 = np.linalg.norm(weights, ord=1) # 6.8 -> sum of |weights|
l2 = np.linalg.norm(weights, ord=2) # 4.32 -> straight-line magnitude
# Ridge-style penalty added to a loss function:
lambda_ = 0.01
ridge_penalty = lambda_ * l2**2
lasso_penalty = lambda_ * l1
print(ridge_penalty, lasso_penalty)- Regularization — Ridge regression penalizes the L2 norm of weights (shrinks everything a little); Lasso penalizes the L1 norm (pushes some weights to exactly zero, giving automatic feature selection).
- Gradient clipping in deep learning caps the L2 norm of the gradient vector before each update step, preventing exploding gradients from destroying training.
- K-nearest-neighbors and clustering algorithms need a norm to define "distance" between points — the choice of norm changes which points count as neighbors.
- City-block routing (literal Manhattan taxicabs, warehouse robots on a grid) is exactly the L1 norm in action — you can't cut diagonally through a building.
- Forgetting to specify
ord=innp.linalg.norm()— it defaults to L2, which silently gives the wrong number if you actually wanted L1. - Applying L2 regularization to bias terms — conventionally, only weights are regularized, not biases, since biases don't contribute to overfitting the same way.
Going deeper
Every valid norm must satisfy three properties: it's zero only for the zero vector, scaling the vector scales the norm proportionally, and it obeys the triangle inequality (going directly somewhere is never longer than going via a detour). These aren't arbitrary rules — they're what make "distance" behave the way our intuition expects.
At the master level: the L0 "norm" (not a true norm, since it fails the scaling property) simply counts the number of non-zero entries — directly optimizing it gives the sparsest possible solution, but the resulting problem is NP-hard. L1 regularization is popular specifically because it's the tightest convex relaxation of L0 — close enough to sparse, but solvable efficiently with standard convex optimization.