Statistical Decision Theory
Expected risk, the Bayes-optimal predictor, and where loss functions come from.
On this page
Beginner: the previous topic (2.1.1) said models are trained by minimizing average loss on the training set. But the number you actually care about is how the model does on new data it hasn't seen — that quantity is called risk. And if you could somehow know the entire true, infinite distribution the data comes from, there would be a single best possible prediction rule for it — not a perfect one, just the best one achievable. That rule is called the Bayes-optimal predictor. No real model ever reaches it exactly (you never have infinite data or the true distribution), but every model is, in a precise sense, trying to approximate it.
Intermediate: training loss is an average over your finite sample; expected risk is the same average taken over the entire true data distribution — training loss is just a noisy, finite-sample estimate of it. The Bayes- optimal predictor f* is whichever function minimizes that true risk, and the risk it achieves, R*, is called the Bayes risk — a hard floor on how well any function could ever do on this problem, no matter how much data or compute you throw at it. It exists because of genuine, irreducible randomness in how y relates to x (two patients with identical measurable symptoms can have different outcomes).
Advanced: here is the idea that ties this whole topic together — what counts as "best" is not fixed; it's entirely determined by which loss function you pick. Under squared loss, the best possible prediction for a given input is the conditional mean of the target. Under absolute loss, it's the conditional median. Under 0-1 loss (classification), it's the most probable class (the mode of the label distribution). These aren't three different approximations of the same target — they are three different targets. Changing your loss function silently changes the statistical quantity your model is even trying to learn, which is exactly why the choice of loss deserves this much scrutiny before you ever pick a model class.
Worked example — one number, by hand: abstractions like "the Bayes-optimal predictor picks the more probable class" are easy to nod along to and hard to actually trust until you've pushed a real number through the machinery once. So take the exact setup the diagram below uses: two classes with equal priors P(y=0) = P(y=1) = 0.5, and class-conditional densities that are both Gaussian with the same spread, σ = 35, but centered at different means — μ₀ = 110 for class 0 and μ₁ = 210 for class 1. Pick a single input value, say x = 140, and ask what the Bayes-optimal predictor does there. First get each class-conditional density at that point (plugging into the normal density formula from the Formula block below): p(x=140 | y=0) ≈ 0.0079, noticeably higher than p(x=140 | y=1) ≈ 0.0015, because 140 sits only 30 units from μ₀ but a full 70 units from μ₁. Weight each by its prior (both 0.5 here, so this step just halves them): joint terms 0.00394 and 0.00077. Bayes' rule says the posterior is each joint term divided by their sum (the "evidence", ≈ 0.00471), which gives P(y=1 | x=140) ≈ 0.163 — about a 16% chance of class 1, hence an 84% chance of class 0. Since 0.163 is nowhere close to crossing one half, the 0-1-loss Bayes-optimal rule derived later in this page confidently predicts class 0 at x = 140. Notice this is the very same x = 140 printed in the Python and C++ code samples further down — run either one and you'll see it print almost exactly 0.163, because that code is doing, in a loop, the identical three-step arithmetic (density → joint → normalize) just performed here by hand. Slide that same arithmetic along the x-axis and the posterior crosses exactly one half at x = 160 — the midpoint of the two means whenever the variances and priors match — which is exactly where the diagram's equal-cost optimum sits, and exactly the boundary the derivation below proves in general.
Expected risk of a predictor f, and the Bayes risk it's measured against:
P is the true (unknown) joint distribution over inputs and labels; L is a loss function measuring how bad a prediction is. The three losses named in the syllabus for this topic:
0-1 loss is for classification (it just counts mistakes); squared and absolute loss are for regression, and they disagree about which errors hurt more — squared loss punishes a miss of 10 units a hundred times harder than a miss of 1 unit, while absolute loss scales the punishment linearly.
Case 1 — regression, squared loss. Fix an input x and ask: what single number f(x) minimizes expected squared error, given everything that's knowable about y at that x? Write m(x) = E[Y | X=x] for the true conditional mean, and add and subtract it inside the square — a completely legal, zero-net-effect algebraic trick:
Expand the square on the right — three terms, no approximation yet:
The middle term is where everything collapses. f(x) - m(x) is just a constant once x is fixed — it doesn't depend on the random variable Y, so it pulls straight out of the expectation. What's left inside, E[Y - m(x) | X=x], is exactly zero by the very definition of m(x) as the conditional mean — a mean is, by construction, the point that deviations average out around. So the whole middle term vanishes, leaving:
The second term doesn't involve f at all — it's pure, irreducible noise in the relationship between x and y. The first term is the only piece you control, and it's a square, so it can never go negative — its minimum possible value, zero, is achieved at exactly one point:
The Bayes-optimal regressor under squared loss is the conditional mean, full stop — and the leftover Var(Y|X=x) at that optimum is precisely the Bayes risk: the error you cannot train away no matter how good your model gets, because it's randomness in the data-generating process itself, not a deficiency of any function f.
Case 2 — binary classification, 0-1 loss. Fix x again, and consider a deterministic guess c ∈ {0, 1}. Its expected 0-1 loss at this x is just the probability the guess is wrong:
Minimizing 1 − P(Y=c|X=x) over the two choices of c is the same as maximizing P(Y=c|X=x) — pick whichever label the data thinks is more probable at this x. Since there are only two classes, P(Y=1|X=x) + P(Y=0|X=x) = 1, so P(Y=1|X=x) is the larger of the two exactly when it exceeds one half:
Where this is used: this is not a coincidence of notation — it is the reason squared-error regressors (linear regression, most neural-net regression heads) are trained the way they are: minimizing squared loss forces the network toward outputting E[Y|X], whether anyone designing it thought about it in those terms or not. It's also exactly why probabilistic classifiers (logistic regression, a softmax head followed by argmax) compare their output probability to 0.5. But look closely at the derivation for case 2 — the 0.5 threshold only fell out because we assumed a false positive and a false negative cost exactly the same amount. Change that assumption — a missed cancer diagnosis is far worse than an unnecessary follow-up test — and the optimal threshold is provably no longer 0.5. That reweighting is called cost-sensitive learning, covered in a later module; the diagram below lets you feel it directly.
Two class-conditional distributions with equal priors and equal variance. The purple dashed line is the true cost-optimal threshold for the cost ratio on the slider; drag your own threshold and watch the expected loss change relative to it.
The from-scratch versions compute the posterior by hand, exactly the way the derivation above says to — no library call hides the Bayes'-rule step. The library version fits on sampled data (not the true parameters) and still recovers essentially the same boundary, which is the whole point: with enough data, a well-matched model finds its way back to the theoretical optimum.
import math
def gaussian_pdf(x, mean, var):
"""N(x; mean, var) -- the class-conditional density p(x | y)."""
coeff = 1.0 / math.sqrt(2.0 * math.pi * var)
return coeff * math.exp(-((x - mean) ** 2) / (2.0 * var))
def posterior(x, params):
"""
params: dict of class label -> (mean, var, prior).
Returns dict of class label -> P(y=class | x), computed explicitly via
Bayes' rule -- no library does this lookup for us:
P(y=c | x) = P(x | y=c) * P(y=c) / sum_c'( P(x | y=c') * P(y=c') )
"""
joint = {c: gaussian_pdf(x, mean, var) * prior for c, (mean, var, prior) in params.items()}
evidence = sum(joint.values())
return {c: j / evidence for c, j in joint.items()}
def bayes_classify(x, params):
"""The Bayes-optimal predictor under 0-1 loss: pick whichever class has the
higher posterior probability (see the derivation in the lesson)."""
post = posterior(x, params)
return max(post, key=post.get)
# Two classes, both Gaussian, with the SAME variance -> the boundary collapses
# to a single point instead of a curve (the "equal-variance" special case).
mean0, mean1 = 110.0, 210.0
var0 = var1 = 35.0 ** 2
prior0 = prior1 = 0.5
params = {
0: (mean0, var0, prior0),
1: (mean1, var1, prior1),
}
for x in (60.0, 110.0, 140.0, 160.0, 172.0, 210.0, 260.0):
post = posterior(x, params)
pred = bayes_classify(x, params)
print("x=%6.1f P(y=1|x)=%.3f predict=%d" % (x, post[1], pred))
# Closed form for the equal-variance, equal-prior case: the two posteriors are
# exactly equal at the midpoint of the two means (derived by setting the two
# weighted densities equal and taking logs -- the quadratic terms cancel
# because the variances are identical, leaving a linear equation in x).
boundary = (mean0 + mean1) / 2.0
print("Analytic Bayes-optimal decision boundary: x* = %.2f" % boundary)- Medical screening. A test that outputs
P(disease | test result)should almost never threshold at 0.5. Suppose a missed cancer case is judged, in expected downstream harm, 9× worse than an unnecessary follow-up biopsy — exactly the cost ratio used in the quiz below. Setting the two cost-weighted expected losses equal (the same algebra behindoptimalThresholdin the diagram) pushes the decision boundary down from 0.5 to roughly a 10% predicted-probability cutoff before the test flags "positive, go biopsy" — deliberately accepting far more false alarms in exchange for catching nearly every real case. - Fraud detection. A card issuer scoring "is this transaction fraudulent" faces two very differently priced mistakes: a missed fraud (false negative) is a direct dollar loss plus a chargeback, while a false positive means declining a legitimate purchase and annoying a real customer. Because fraud is also rare (a heavily skewed prior, not just an asymmetric cost), issuers score every transaction and set the alert threshold low enough to catch most fraud while keeping the false-positive rate tolerable — then route the ambiguous middle band to a human reviewer rather than trusting either raw class of decision alone.
- Spam filtering. The asymmetry runs the other way from medical screening: sending a real client email to spam (a false positive on the "is spam" label) is usually judged worse than letting one extra spam message through (a false negative), so production filters often push the spam threshold above 0.5 rather than below it — the same math as the biopsy example, just with the cost ratio flipped to favor the opposite error.
- Insurance underwriting. An insurer estimating
P(claim | applicant features)doesn't threshold that probability into a binary accept/reject at all in most lines of business — it feeds an explicit actuarial cost matrix (expected claim payout vs. lost premium revenue from declining a profitable policy) to set both the accept/decline boundary and the premium charged in the gray zone, which is the cost-sensitive framework from this lesson applied continuously rather than at one fixed cutoff. - Hiring and credit decisions — and their fairness cost. A resume screener or a loan-approval model that thresholds a predicted score at whatever value minimizes aggregate expected cost can still produce very different false-negative rates across subgroups if the underlying class-conditional distributions (or the base rates) differ by group — the same asymmetric-cost machinery that correctly favors sensitivity in medical screening can just as easily encode and launder discrimination if the "cost" being minimized was never audited for whose errors it's willing to tolerate. This is precisely why fairness-aware ML treats the decision threshold, not just the model's scores, as something requiring its own scrutiny.
- Industrial quality control. A vision model deciding whether a manufactured part passes inspection faces a stark cost asymmetry: shipping one defective brake component (a false negative) can mean a recall and lawsuits, while scrapping one good part (a false positive) costs only that part's material and labor. With a cost ratio that can run into the hundreds, the optimal threshold sits far below 0.5 — the line will happily flag 30% of good parts for human re-inspection if that is what it takes to drive the false-negative rate near zero.
- Weather forecasting. "70% chance of rain" is not trying to be a single best point guess minimizing some loss — it is a probability meant to be calibrated (it should rain on roughly 70% of days that get that forecast). Point predictions born from a loss function and calibrated probabilities are related but distinct goals, and conflating them is a common source of confusion when reading model output.
- Treating 0.5 as a universal, loss-function-blessed decision threshold. It is only optimal under 0-1 loss with equal costs for both error types — as soon as false positives and false negatives cost different amounts, the derivation above shows the optimal threshold provably moves.
- Conflating the Bayes-optimal predictor (a theoretical ideal defined using the true, unknowable data distribution) with "the best model I can actually fit." Any real model is also constrained by its hypothesis class and by having only a finite training sample — the gap between what you can achieve and the Bayes risk is exactly what the next topic, 2.1.3 Bias-Variance Trade-off, breaks apart into named, separately-addressable pieces.
- Reaching for squared loss by default without asking what you actually want estimated. If your target variable has a long tail or outliers, the conditional mean (what squared loss targets) can be dragged far from where most of the data actually sits, while the conditional median (what absolute loss targets) stays put — this is precisely the motivation for quantile regression, covered in a later module.
Going deeper
Every loss function silently names a target statistic of Y | X=x: squared loss names the mean, absolute loss names the median, 0-1 loss names the mode. This is not a coincidence restricted to these three — it generalizes completely. The pinball loss (also called quantile loss), parameterized by a quantile level τ ∈ (0, 1), is asymmetric: it penalizes over-prediction and under-prediction by different amounts depending on τ, and its Bayes- optimal predictor is exactly the τ-th conditional quantile of Y | X=x. Absolute loss is the special case τ = 0.5 (the median). This is how models that output prediction intervals — "80% of the time, the true value falls between these two numbers" — are trained: fit two quantile regressors at τ = 0.1 and τ = 0.9, and the gap between them is your interval.
A hospital's screening classifier outputs a calibrated P(disease | test result). Missing a true case is judged 9× worse than a false alarm. Should the hospital still flag a patient as positive only when this probability exceeds 0.5?
No. Thresholding at 0.5 is only Bayes-optimal under 0-1 loss with equal costs for both error types. With a 9:1 cost ratio favoring catching true cases, the same derivation used in this lesson (setting the two cost-weighted expected losses equal) pushes the optimal threshold below 0.5 — the hospital should flag positive at a lower probability cutoff so it stops missing as many real cases, accepting more false alarms as the deliberate trade-off. The interactive diagram's cost slider shows exactly this shift.
Risk is loss averaged over the true data distribution, not the training set; the Bayes- optimal predictor is the best any function could ever do against that risk, and it is a different mathematical object — mean, median, or mode of Y|X — depending entirely on which loss you chose. Every later model in this chapter is, underneath its specific machinery, an attempt to approximate one of these three targets from finite, noisy data.