Gaussian Discriminant Analysis — LDA / QDA
Modeling each class as a Gaussian — and the one choice that makes the boundary linear or curved.
On this page
Beginner: Naive Bayes (section 2.5.1) modeled each feature of a class separately and multiplied the pieces together. Gaussian Discriminant Analysis (GDA) makes a different simplifying assumption: instead of treating features as independent, it says that within each class, the whole feature vector looks like a single bump of probability mass shaped like a (possibly tilted, possibly stretched) ellipse — a multivariate Gaussian. To classify a new point, GDA asks two very intuitive questions for each class: "how plausible is this point under class k's Gaussian cloud?" and "how common is class k in general?" — then multiplies those two answers together (that's Bayes' rule) and picks whichever class scores highest. The one modeling decision that this lesson is entirely built around is deceptively small: do all the classes' ellipses share the same shape and tilt, or does each class get its own? Sharing gives you LDA (Linear Discriminant Analysis); letting them differ gives you QDA (Quadratic Discriminant Analysis) — and that single choice is the entire difference between a straight decision boundary and a curved one.
Intermediate: picture two overlapping clouds of points, one per class. If both clouds have the same spread and orientation — just centered in different places — then the natural dividing line between "closer to cloud A" and "closer to cloud B," measured in a way that accounts for that shared spread, is a straight line exactly halfway between the two centers. That's LDA, and it's cheap: you only ever have to estimate ONE covariance matrix from all the data pooled together. If instead the two clouds are different shapes — one long and thin, the other round, tilted differently — a single straight line can no longer separate them well; the natural boundary has to curve to hug whichever class's ellipse is "tighter" in a given direction. That's QDA, and the price for that extra flexibility is a separate covariance matrix per class, which needs proportionally more data per class to estimate reliably before it starts helping rather than hurting.
Advanced: the two-class, shared-covariance case (LDA) is not just "linear like logistic regression" as a vague description — Derivation 5 below shows the two are algebraically identical: GDA's posterior, once you substitute Gaussian densities into Bayes' rule and simplify, reduces to exactly σ(θᵀx + b) for a θ and b written explicitly in terms of μ₀, μ₁, and Σ. Section 2.4.1's logistic regression fits that same sigmoid form directly, with no distributional assumption at all, by discriminative maximum likelihood on the posterior itself. LDA reaches the identical functional form indirectly, via generative maximum likelihood on μ, Σ, and π. Same hypothesis, two different roads to it — and which road is better depends entirely on whether the Gaussian assumption is actually true, which is precisely the question section 2.5.4 answers rigorously.
The class-conditional density every discriminant function below is built from — the multivariate Gaussian, dimension d:
LDA's discriminant function (shared Σ across classes) — linear in x:
QDA's discriminant function (own Σₖ per class) — quadratic in x:
In both cases, classify x as argmax_k δ_k(x). Every step from the Gaussian density to these two formulas — and from LDA's formula to an exact logistic sigmoid — is derived from scratch below.
Each class k gets two parameters: a mean vector μₖ ∈ ℝᵈ (where that class's data is centered) and a covariance matrix Σₖ ∈ ℝ^{d×d} (how that data is spread out and correlated around the mean). The density itself,
is entirely governed by the quantity inside the exponent, (x−μ)ᵀΣ⁻¹(x−μ), called the squared Mahalanobis distance from x to μ. Points sharing the same Mahalanobis distance from μ form the level sets of this density — and because Σ is symmetric positive definite (see section 1.15's Positive Definite & Cholesky lesson for exactly what that guarantees and why it's guaranteed for a genuine covariance matrix), those level sets are always closed ellipsoids centered at μ, never open or unbounded curves.
Concretely, section 1.15's eigendecomposition machinery gives the ellipsoid's exact shape: the eigenvectors of Σ are the ellipse's principal axis directions, and the square roots of the corresponding eigenvalues are how far the ellipse extends along each of those axes. A covariance with equal eigenvalues and zero off-diagonal entries produces a plain circle (isotropic spread, no preferred direction); unequal eigenvalues stretch it into an ellipse (one direction more variable than the other); a nonzero off-diagonal entry (correlated features) tilts that ellipse's axes away from the coordinate axes entirely. Every ellipse drawn in the diagrams below is built exactly this way — sampled directly from Σ's eigenvectors and eigenvalues, not approximated.
Where this is used: this geometric picture is the entire visual language of this lesson. LDA forces every class's ellipse to have the same shape and tilt (same eigenvectors and eigenvalues, just re-centered); QDA lets each class's ellipse be a completely independent shape. Diagram 2 below shows precisely this contrast on one fixed dataset.
GDA follows the exact same generative recipe as Naive Bayes (section 2.5.1) — the only thing that changes is which family of class-conditional density gets plugged in. Model a prior over classes, P(y=k) = π_k (with Σₖπₖ=1), and a class-conditional density, P(x|y=k) = 𝒩(x;μₖ,Σₖ) — "if I generated a point from class k, what would it look like." Bayes' rule then combines them into the quantity that's actually needed for classification, the posterior:
Because the denominator is the same number for every class k, it never affects which class has the largest posterior — so classification only ever needs the numerator, and (working in log-space exactly as section 2.5.1 does, for the same numerical-stability reasons) only needs log πₖ + log 𝒩(x;μₖ,Σₖ) compared across classes. That comparison quantity is exactly what "the discriminant function δₖ(x)" means for the rest of this lesson.
Fitting πₖ, μₖ, and Σₖ is plain maximum likelihood (section 2.1.6's MLE principle), and for a Gaussian family it has closed-form answers — no iterative solver needed at all. With Nₖ training points in class k out of N total:
For the covariance, QDA estimates one matrix per class from that class's own points:
while LDA pools every class's centered points into one shared estimate — a weighted average of the per-class covariances above:
Where this is used: these are exactly the formulas the "from scratch" code below implements — mean and (pooled or per-class) covariance from the training data, fed straight into δₖ(x). The rest of this lesson is entirely about what happens to that discriminant function's shape depending on which covariance estimate — pooled or per-class — gets used.
Compare any two classes k and j by taking the log of their posterior ratio. The Bayes'-rule denominator is identical for both classes, so it cancels immediately:
Substitute the Gaussian density (using the SAME shared Σ for both classes — the one modeling choice LDA makes) and take the log of the ratio directly:
The normalizing constant (2π)^{d/2}|Σ|^{1/2} and the −½log|Σ| term are identical for both classes (same Σ) and cancel exactly, leaving:
Now expand each quadratic form — (x−μ)ᵀΣ⁻¹(x−μ) = xᵀΣ⁻¹x − 2xᵀΣ⁻¹μ + μᵀΣ⁻¹μ (using Σ⁻¹'s symmetry to combine the cross terms) — and subtract the two expansions. This is the key step: the xᵀΣ⁻¹x term is identical in both expansions, because both classes are using the exact same Σ⁻¹ — so it cancels completely:
What's left is linear in x — the quadratic term that would have made the boundary curve is gone. Substituting back and collecting every term that depends on class k into one function δₖ(x), and every term that depends on class j into δⱼ(x):
exactly the LDA discriminant from the Formula section. Classification is argmax_k δₖ(x), and the boundary between any two classes, δₖ(x)=δⱼ(x), is a single linear equation in x — a hyperplane. Diagram 1 below draws exactly this hyperplane (a line, in 2D) and lets you watch it move as μₖ moves.
Repeat the identical derivation, but now each class keeps its own covariance Σₖ. The normalizing constants no longer match, so −½log|Σₖ| no longer cancels between classes, and — critically — neither does the quadratic term, because Σₖ⁻¹ ≠ Σⱼ⁻¹ in general:
Expanding as before, the bracketed term now contains xᵀΣₖ⁻¹x − xᵀΣⱼ⁻¹x = xᵀ(Σₖ⁻¹ − Σⱼ⁻¹)x — a genuine quadratic form in x that survives, because there's no longer a common Σ⁻¹ for it to cancel against. The per-class discriminant function keeps its own quadratic term explicitly:
The boundary δₖ(x)=δⱼ(x) is now a genuine second-degree equation in x — a conic section. Which conic (ellipse, parabola, or hyperbola) depends on the eigenvalues of Σₖ⁻¹ − Σⱼ⁻¹: definite (all same-signed eigenvalues) gives an ellipse-shaped boundary piece, indefinite (mixed-sign eigenvalues) gives a hyperbola with two open branches, and the degenerate borderline case gives a parabola. Diagram 2 below traces this curve exactly, root by root, from the discriminant functions above — not sketched by hand.
The direct bias-variance trade-off (cross-reference section 2.1.3): QDA's extra flexibility is not free. A shared covariance matrix has d(d+1)/2 free parameters to estimate, period — LDA pools all the training data (across every class) into that one estimate. QDA needs K separate d(d+1)/2- parameter covariance matrices, each estimated from only that one class's data. QDA can fit more varied class shapes (lower bias, when classes genuinely have different covariance structure) but each of its K covariance estimates is noisier, coming from a fraction of the data LDA gets to use (higher variance) — and in high dimension with modest per-class sample sizes, this is exactly what makes QDA's covariance estimates unstable (see the Pitfall below).
Specialize Derivation 3 to two classes, k=1 and j=0. The log-odds derived there,
expand out fully using δₖ(x) = xᵀΣ⁻¹μₖ − ½μₖᵀΣ⁻¹μₖ + log πₖ from Derivation 3:
The middle term simplifies nicely: because Σ⁻¹ is symmetric, μ₁ᵀΣ⁻¹μ₀ = μ₀ᵀΣ⁻¹μ₁, so those two cross terms cancel when you expand (μ₁−μ₀)ᵀΣ⁻¹(μ₁+μ₀) — which means μ₁ᵀΣ⁻¹μ₁ − μ₀ᵀΣ⁻¹μ₀ = (μ₁−μ₀)ᵀΣ⁻¹(μ₁+μ₀) exactly. Substituting and defining θ = Σ⁻¹(μ₁−μ₀) (so the first term is just θᵀx):
with b = −½θᵀ(μ₁+μ₀) + log(π₁/π₀). This says the log-odds is exactly linear in x — and "log-odds is linear in x" is, by definition (recall the logit derivation from section 2.4.1), the same statement as "the posterior is a sigmoid of that linear score":
This is not an approximation or a resemblance — it's an algebraic identity. Diagram 3 below plugs in a concrete μ₀, μ₁, Σ, computes both sides completely independently (the raw Bayes'-rule posterior from the two Gaussian densities, and the sigmoid of θᵀx+b using the θ, b formulas just derived), and the two curves land on top of each other to six decimal places.
The precise conclusion. LDA and logistic regression fit the exact same functional form for the posterior — a sigmoid of a linear score. What differs is how θ and b get chosen. Logistic regression (section 2.4.1) estimates them directly, by discriminative MLE that maximizes the likelihood of P(y|x) alone — it makes no assumption whatsoever about what P(x|y) looks like. LDA arrives at values of θ and b only indirectly, as a byproduct of generative MLE that fits μ, Σ, and π to make the Gaussian class-conditionals fit well — a strictly stronger assumption. That means LDA's hypothesis class is a strict subset of what logistic regression can represent: every LDA solution is some logistic regression solution, but not every logistic-regression-fittable boundary comes from any valid choice of Gaussian μ₀, μ₁, Σ. When the Gaussian assumption genuinely holds, that extra structure is free information, and LDA converges to the right boundary from less data than logistic regression needs; when it doesn't hold, logistic regression's weaker, boundary-only assumption makes it the more robust choice. Section 2.5.4 turns this exact trade-off into a rigorous asymptotic result.
Both classes share one covariance shape (just re-centred). Drag the slider to move class 1's mean μ₁ and watch the straight decision boundary slide and rotate to stay exactly halfway between the means, in the Σ-weighted (Mahalanobis) sense.
Same 68 seeded points throughout. Toggle the covariance assumption applied to them — sharing one pooled Σ (LDA) yields a straight boundary; letting each class keep its own Σₖ (QDA) yields a curved one, traced exactly from the discriminant functions derived above.
Scrubbing along the line connecting μ₀ and μ₁, the raw Bayes'-rule posterior from the two Gaussians (violet) and a plain sigmoid using the derived θ, b (dashed) coincide everywhere to numerical precision.
All three implementations estimate μₖ, Σ (pooled) or Σₖ (per-class), and πₖ exactly as derived above, then classify by the discriminant function that reduces to LDA's linear form when the covariance is shared and to QDA's quadratic form when it isn't.
#include <cmath>
#include <iostream>
#include <vector>
struct Vec2 { double x, y; };
struct Mat2 { double a, b, c, d; }; // [[a,b],[c,d]]
Mat2 inv2(const Mat2& m) {
double det = m.a * m.d - m.b * m.c;
return { m.d / det, -m.b / det, -m.c / det, m.a / det };
}
double det2(const Mat2& m) { return m.a * m.d - m.b * m.c; }
Vec2 matVec(const Mat2& m, const Vec2& v) {
return { m.a * v.x + m.b * v.y, m.c * v.x + m.d * v.y };
}
double dot(const Vec2& u, const Vec2& v) { return u.x * v.x + u.y * v.y; }
Vec2 sub(const Vec2& u, const Vec2& v) { return { u.x - v.x, u.y - v.y }; }
Vec2 mean(const std::vector<Vec2>& pts) {
Vec2 m{0, 0};
for (const auto& p : pts) { m.x += p.x; m.y += p.y; }
m.x /= pts.size(); m.y /= pts.size();
return m;
}
Mat2 covariance(const std::vector<Vec2>& pts, const Vec2& mu) {
Mat2 c{0, 0, 0, 0};
for (const auto& p : pts) {
double dx = p.x - mu.x, dy = p.y - mu.y;
c.a += dx * dx; c.b += dx * dy; c.c += dx * dy; c.d += dy * dy;
}
double n = static_cast<double>(pts.size());
return { c.a / n, c.b / n, c.c / n, c.d / n };
}
// General discriminant -- reduces exactly to the LDA form when both classes share a covariance.
double discriminant(const Vec2& x, const Vec2& mu, const Mat2& cov, double prior) {
Mat2 covInv = inv2(cov);
double logDet = std::log(det2(cov));
Vec2 diff = sub(x, mu);
double quad = dot(diff, matVec(covInv, diff));
return -0.5 * logDet - 0.5 * quad + std::log(prior);
}
int main() {
std::vector<Vec2> class0 = {{-2.1,-1.4},{-1.6,0.1},{-0.9,-0.8},{-1.8,-0.3},{-1.2,-1.1}};
std::vector<Vec2> class1 = {{1.4,0.6},{2.0,1.3},{0.9,1.7},{1.7,0.4},{2.3,0.9}};
Vec2 mu0 = mean(class0), mu1 = mean(class1);
Mat2 cov0 = covariance(class0, mu0), cov1 = covariance(class1, mu1);
double n0 = class0.size(), n1 = class1.size(), n = n0 + n1;
Mat2 pooled = {
(cov0.a * n0 + cov1.a * n1) / n, (cov0.b * n0 + cov1.b * n1) / n,
(cov0.c * n0 + cov1.c * n1) / n, (cov0.d * n0 + cov1.d * n1) / n,
};
double prior0 = n0 / n, prior1 = n1 / n;
Vec2 test{0.2, 0.0};
double ldaScore0 = discriminant(test, mu0, pooled, prior0);
double ldaScore1 = discriminant(test, mu1, pooled, prior1);
double qdaScore0 = discriminant(test, mu0, cov0, prior0);
double qdaScore1 = discriminant(test, mu1, cov1, prior1);
std::cout << "LDA predicts class " << (ldaScore1 > ldaScore0 ? 1 : 0) << "\n";
std::cout << "QDA predicts class " << (qdaScore1 > qdaScore0 ? 1 : 0) << "\n";
return 0;
}- Face recognition — Fisherfaces. LDA's use here is a genuinely different role than direct classification: it's applied as a supervised dimensionality-reduction step (Fisherfaces), finding the projection that best separates known identities before a simpler classifier runs on the reduced space — a preview of the projection techniques in section 2.12.
- Medical diagnosis from biomarkers. Continuous physiological measurements (blood chemistry panels, imaging-derived measurements) are frequently close enough to Gaussian per diagnostic class that GDA gives a strong, fast, interpretable baseline — and unlike a black-box model, its fitted μₖ and Σₖ are themselves clinically readable summaries of "what a typical patient in this category looks like."
- Credit scoring and risk tiers. LDA has been a staple in credit-risk modeling for decades: fast to fit in closed form, well-calibrated when its assumptions roughly hold, and its linear score has the same auditable "one coefficient per feature" interpretability that regulated finance already expects from logistic regression.
- Quality control / manufacturing defect classification. Sensor measurements clustered around a "normal operating" mean with roughly elliptical noise are a natural fit for a Gaussian class-conditional, and QDA's flexibility helps when a defect class genuinely has a different noise shape, not just a different mean.
- A fast, well-calibrated baseline in general. Because both μ and Σ have closed-form MLE solutions, GDA trains near-instantly even on large feature sets — it remains a standard first thing to try before reaching for an iterative discriminative model, precisely to see whether the Gaussian generative story already explains most of the separation.
- Fitting QDA with too little data per class. Each of the
Kper-class covariance matrices needs enough points to estimated(d+1)/2parameters reliably. With few points relative tod,Σ̂ₖcan become singular or near-singular — its inverse blows up, log-likelihoods and discriminant scores become numerically unstable, and the fitted boundary can swing wildly on data that barely changed. This is the direct, concrete consequence of the bias-variance trade-off in Derivation 4 — it isn't a bug, it's what "higher variance" looks like in practice, and the ExpertNote below covers the standard fix. - Assuming the Gaussian class-conditional assumption when it visibly isn't true. Heavily skewed features (e.g. income, wait times), multimodal classes (a "class" that's secretly two different sub-populations), or hard categorical/count features are all poor fits for a single ellipsoidal Gaussian — GDA's boundary can be badly wrong in exactly the regions where the Gaussian shape and the true data shape disagree most. As the module overview notes, "wrong but useful" often still applies to the resulting boundary — but it's worth actually checking (histograms, QQ-plots) rather than assuming.
- Conflating "Linear Discriminant Analysis" the classifier with LDA as a dimensionality-reduction technique. The name is genuinely overloaded: this lesson's LDA (a generative Bayes'-rule classifier) and the unsupervised-adjacent "project onto the directions that best separate known classes" LDA used for dimensionality reduction (used inside Fisherfaces, section 2.12) share their derivation machinery but answer different questions — one outputs a class label, the other outputs a lower-dimensional coordinate.
Going deeper
Regularized / shrinkage discriminant analysis. The QDA instability flagged in the Pitfall above has a standard, practical fix: instead of committing fully to either the pooled estimate Σ̂ (LDA) or the fully separate estimates Σ̂ₖ (QDA), blend them with a shrinkage parameter α ∈ [0,1]:
At α=0 this is plain QDA; at α=1 it's plain LDA; intermediate values interpolate continuously between them, letting each class keep some of its own covariance structure while borrowing statistical strength from the pooled estimate to stay numerically well-conditioned with limited per-class data. This is a specific instance of the general covariance-shrinkage idea covered in full in section 2.13.11 — the same bias-variance logic (a little bias from shrinking toward a simpler, pooled estimate, in exchange for a large reduction in variance) that motivated ridge regression's penalty (section 2.3.2) shows up again here, applied to a covariance matrix instead of a coefficient vector.
Separately, it's worth knowing that LDA has a second, entirely different historical derivation: Fisher's original 1936 formulation never mentions Gaussians or Bayes' rule at all. It asks a purely geometric question — find the projection direction w that maximizes the ratio of between-class scatter to within-class scatter, wᵀSᵦw / wᵀS_ww — and it's a genuinely remarkable fact that, for the two-class, shared-covariance case, this scatter-maximizing direction comes out proportional to the exact same θ = Σ⁻¹(μ₁−μ₀) derived from Bayes' rule above. Two completely different starting philosophies — one probabilistic, one geometric — landing on the identical answer is a strong signal that θ is capturing something real about the structure of the problem, not an artifact of either derivation's particular assumptions.
Why is LDA's decision boundary always linear while QDA's is generally quadratic — trace the answer back to the covariance assumption in one sentence?
Both boundaries come from setting δ_k(x)=δ_j(x), and each δ contains an x^T Σ_k^{-1} x quadratic term; LDA forces every class to share the identical Σ (so Σ_k^{-1}=Σ_j^{-1}), which makes that quadratic term cancel exactly when the two δ's are subtracted, leaving only linear terms in x -- whereas QDA lets each class keep its own Σ_k, so Σ_k^{-1}≠Σ_j^{-1} in general and the quadratic term x^T(Σ_k^{-1}-Σ_j^{-1})x survives the subtraction, producing a genuinely quadratic (conic-section) boundary.
GDA's entire story reduces to one modeling choice with two names: share Σ across classes and you get LDA's linear boundary (and, in the two-class case, an exact algebraic match to logistic regression's sigmoid); let each class keep its own Σₖ and you get QDA's curved boundary, at the cost of estimating far more covariance parameters from the same data. Both are still fully parametric — they commit to the Gaussian shape before ever looking at how the data is actually spread out. Section 2.5.3, k-Nearest Neighbors, drops that parametric assumption entirely: instead of fitting μ, Σ, and π to a chosen density family, it estimates the posterior directly and non-parametrically from whichever training points happen to be nearby.