Quadratic Forms, Convexity & the Hessian
Why some loss landscapes are easy (convex) and most deep learning isn't.
On this page
Beginner: a quadratic form is an expression built entirely from squared and cross terms, like f(x,y) = ax² + dy² (or, more generally, xᵀAx). Its graph is a curved bowl-, dome-, or saddle-shaped surface — exactly the shapes that loss functions and optimization landscapes are made of.
Intermediate: whether that surface is a bowl (has one clear minimum), a dome (one clear maximum), or a saddle (curves up in one direction and down in another, with no true min or max) is determined entirely by the sign pattern of A's eigenvalues (section 1.7): all positive → bowl (A is positive definite, section 1.15); all negative → dome; mixed signs → saddle.
Advanced: a function is convex if its curvature (captured by its Hessian matrix of second derivatives — a matrix, exactly like the A in a quadratic form) is positive semi-definite everywhere. Convexity is the single property that guarantees gradient descent finds the global minimum, not just a nearby local one — which is precisely why linear regression and logistic regression always train reliably, while deep neural networks (whose loss landscapes are emphatically not convex) can get stuck.
At a critical point (gradient = 0), the Hessian's sign pattern classifies it: positive definite → local minimum, negative definite → local maximum, indefinite (mixed signs) → saddle point.
The second-order Taylor expansion of any smooth function f around a point x, for a small step h, is:
At a critical point, ∇f(x) = 0 by definition — the linear term vanishes entirely, leaving:
— exactly the quadratic form this lesson studies, now with A replaced by the Hessian H. If H is positive definite, this is positive for every nonzero step h, meaning f(x+h) > f(x) in every direction — a genuine local minimum. If H is negative definite, every step decreases f — a local maximum. If H is indefinite, some directions increase f and others decrease it from the very same point — a saddle.
Where this is used: this is the rigorous version of the "second derivative test" from single-variable calculus, generalized to as many dimensions as a neural network has parameters — and it's exactly what Newton's method and K-FAC-style optimizers use the Hessian for.
Drag the sliders through zero — watch the contours flip from ellipses (convex) to hyperbola-like asymptotes (saddle) exactly when a sign changes.
This six-line function is exactly the "second derivative test" from calculus, generalized to many dimensions using nothing but eigenvalues.
#include <cstdio>
#include <cmath>
#include <array>
#include <string>
// Closed-form eigenvalues of a symmetric 2x2 matrix -- no linear-algebra library.
std::array<double, 2> eigenvaluesSymmetric2x2(const std::array<std::array<double, 2>, 2>& H) {
double a = H[0][0], b = H[0][1];
double c = H[1][0], d = H[1][1];
double mean = (a + d) / 2.0;
double diff = (a - d) / 2.0;
double disc = std::sqrt(diff * diff + b * c);
return {mean - disc, mean + disc};
}
std::string classify(const std::array<std::array<double, 2>, 2>& H) {
auto eig = eigenvaluesSymmetric2x2(H);
if (eig[0] > 0 && eig[1] > 0) return "local minimum (convex bowl)";
if (eig[0] < 0 && eig[1] < 0) return "local maximum (concave dome)";
return "saddle point";
}
int main() {
std::printf("%s\n", classify({{{2, 0}, {0, 3}}}).c_str()); // local minimum
std::printf("%s\n", classify({{{2, 0}, {0, -3}}}).c_str()); // saddle point
std::printf("%s\n", classify({{{-2, 0}, {0, -3}}}).c_str()); // local maximum
return 0;
}- Linear and logistic regression have provably convex loss functions — this is why they always converge to the same, globally optimal answer regardless of initialization.
- Support Vector Machines are formulated specifically as convex optimization problems, which is a large part of why they were so reliable before deep learning became dominant.
- Newton's method in optimization uses the Hessian directly (
x ← x − H⁻¹∇f) to take smarter steps than plain gradient descent, converging much faster near a minimum — at the cost of needing to compute and invert the Hessian. In practice, that inversion is done approximately via conjugate gradient (section 1.34), using Hessian-vector products (section 1.32) so the full Hessian is never actually formed.
- Assuming a deep learning loss surface is convex — it almost never is; training relies on good initialization, architecture choices, and optimizers robust to non-convexity, not on convexity guarantees.
- Confusing "gradient is zero" with "this is a minimum" — a zero gradient only means a critical point; the Hessian's sign pattern is what actually classifies it.
Going deeper
For genuinely convex problems, any local minimum is automatically the global minimum — this single fact is why convex optimization is considered a "solved" field with strong theoretical guarantees, while general non-convex optimization (most of deep learning) is not.
At the master level: a striking, counter-intuitive result from high-dimensional random matrix theory (section 1.19) is that in very high-dimensional non-convex landscapes, critical points with a mix of positive and negative Hessian eigenvalues (saddle points) vastly outnumber true local minima — and most of the "getting stuck" behavior once blamed on bad local minima in deep learning is now understood to be about escaping saddle points instead.