Quadratic Forms, Convexity & the Hessian
Why some loss landscapes are easy (convex) and most deep learning isn't.
On this page
Beginner: a quadratic form is an expression built entirely from squared and cross terms, like f(x,y) = ax² + dy² (or, more generally, xᵀAx). Its graph is a curved bowl-, dome-, or saddle-shaped surface — exactly the shapes that loss functions and optimization landscapes are made of.
Intermediate: whether that surface is a bowl (has one clear minimum), a dome (one clear maximum), or a saddle (curves up in one direction and down in another, with no true min or max) is determined entirely by the sign pattern of A's eigenvalues (section 1.7): all positive → bowl (A is positive definite, section 1.15); all negative → dome; mixed signs → saddle.
Advanced: a function is convex if its curvature (captured by its Hessian matrix of second derivatives — a matrix, exactly like the A in a quadratic form) is positive semi-definite everywhere. Convexity is the single property that guarantees gradient descent finds the global minimum, not just a nearby local one — which is precisely why linear regression and logistic regression always train reliably, while deep neural networks (whose loss landscapes are emphatically not convex) can get stuck.
At a critical point (gradient = 0), the Hessian's sign pattern classifies it: positive definite → local minimum, negative definite → local maximum, indefinite (mixed signs) → saddle point.
The second-order Taylor expansion of any smooth function f around a point x, for a small step h, is:
At a critical point, ∇f(x) = 0 by definition — the linear term vanishes entirely, leaving:
— exactly the quadratic form this lesson studies, now with A replaced by the Hessian H. If H is positive definite, this is positive for every nonzero step h, meaning f(x+h) > f(x) in every direction — a genuine local minimum. If H is negative definite, every step decreases f — a local maximum. If H is indefinite, some directions increase f and others decrease it from the very same point — a saddle.
Where this is used: this is the rigorous version of the "second derivative test" from single-variable calculus, generalized to as many dimensions as a neural network has parameters — and it's exactly what Newton's method and K-FAC-style optimizers use the Hessian for.
Drag the sliders through zero — watch the contours flip from ellipses (convex) to hyperbola-like asymptotes (saddle) exactly when a sign changes.
This six-line function is exactly the "second derivative test" from calculus, generalized to many dimensions using nothing but eigenvalues.
import numpy as np
def classify(H):
eigvals = np.linalg.eigvalsh(H) # eigvalsh: for symmetric matrices
if np.all(eigvals > 0):
return "local minimum (convex bowl)"
if np.all(eigvals < 0):
return "local maximum (concave dome)"
return "saddle point"
print(classify(np.array([[2, 0], [0, 3]]))) # local minimum
print(classify(np.array([[2, 0], [0, -3]]))) # saddle point
print(classify(np.array([[-2, 0], [0, -3]]))) # local maximum- Linear and logistic regression have provably convex loss functions — this is why they always converge to the same, globally optimal answer regardless of initialization.
- Support Vector Machines are formulated specifically as convex optimization problems, which is a large part of why they were so reliable before deep learning became dominant.
- Newton's method in optimization uses the Hessian directly (
x ← x − H⁻¹∇f) to take smarter steps than plain gradient descent, converging much faster near a minimum — at the cost of needing to compute and invert the Hessian. In practice, that inversion is done approximately via conjugate gradient (section 1.34), using Hessian-vector products (section 1.32) so the full Hessian is never actually formed.
- Assuming a deep learning loss surface is convex — it almost never is; training relies on good initialization, architecture choices, and optimizers robust to non-convexity, not on convexity guarantees.
- Confusing "gradient is zero" with "this is a minimum" — a zero gradient only means a critical point; the Hessian's sign pattern is what actually classifies it.
Going deeper
For genuinely convex problems, any local minimum is automatically the global minimum — this single fact is why convex optimization is considered a "solved" field with strong theoretical guarantees, while general non-convex optimization (most of deep learning) is not.
At the master level: a striking, counter-intuitive result from high-dimensional random matrix theory (section 1.19) is that in very high-dimensional non-convex landscapes, critical points with a mix of positive and negative Hessian eigenvalues (saddle points) vastly outnumber true local minima — and most of the "getting stuck" behavior once blamed on bad local minima in deep learning is now understood to be about escaping saddle points instead.