Bias–Variance Trade-off
Decomposing prediction error into bias, variance, and irreducible noise.
On this page
Beginner: imagine three people shooting arrows at a dartboard, trying to hit the true bullseye. One person's scope is misaligned — their arrows land in a tight little cluster, but that cluster sits consistently off to one side. That's bias: consistently missing in the same direction. A second person's scope is fine, but their hands shake — arrows scatter widely all around the bullseye, sometimes close, sometimes far, with no consistent direction to the miss. That's variance: inconsistency from one shot to the next. Even a perfect archer with a perfect scope and a perfectly steady hand still has to contend with irreducible error — a gust of wind on any given shot that nobody could have predicted or corrected for in advance.
Intermediate: translate the analogy into modeling. "Bias" here does not mean any one fitted model looks wrong on the data it was trained on — it means: if you imagine drawing many different training sets from the same underlying distribution and fitting your model class fresh on each one, the average prediction across all of those fits, at a given point x, is systematically off from the true function's value at x. "Variance" means: that same prediction at x swings around a lot depending on which particular training set you happened to draw — retrain on a slightly different sample and you get a noticeably different answer. Both quantities are properties of the estimator — the model class and fitting procedure — not properties of any single dataset or any single fitted model.
Advanced: this is exactly why model complexity matters. High-capacity models — deep, unpruned decision trees; high-degree polynomials; heavily overparameterized networks — can bend to fit almost any training set closely, so their average fit across many training sets tends to land close to the truth (low bias), but exactly how they bend is highly sensitive to the specific noise in whichever training set they saw (high variance). Simple models — a straight line, a shallow tree — can't bend much no matter what data they see, so they're stable across training sets (low variance) but systematically miss curvature in the true function they're too rigid to represent (high bias). The next lesson on overfitting, underfitting, and capacity is the empirical, train/test-curve view of this exact same phenomenon — this lesson is its theoretical backbone.
Working the two extremes concretely: the toy curves driving the diagram below make this arithmetic, not just intuition. They're built from Bias²(c) = 9/c² and Variance(c) = 0.05·c² for a complexity dial c running from 1 (rigid) to 10 (extremely flexible), plus a fixed noise floor of 0.3. At c = 1, the simplest setting, bias² alone is 9/1² = 9 while variance is a negligible 0.05·1² = 0.05 — total error of roughly 9.35 is almost entirely bias: the model is so rigid that its systematic miss from the true function swamps everything else, and refitting it on a different sample barely changes the answer at all. At c = 10, the most flexible setting, the roles flip completely: bias² has collapsed to 9/100 = 0.09 (the model is now expressive enough that its average fit is essentially correct) but variance has ballooned to 0.05·100 = 5 — total error of about 5.39 is almost entirely variance: the fit swings wildly depending on which particular training sample it happened to see. Neither extreme's total error is anywhere close to the interior minimum, which is exactly the point — the U-shape isn't a metaphor here, it's what you get from adding a strictly decreasing curve to a strictly increasing one and finding where their sum bottoms out (here, at c≈3.66 exactly).
Two named algorithms at opposite ends: a decision stump (a tree with a single split, i.e. depth 1) can only ever ask one yes/no question of the data before predicting, so no matter how the training sample is drawn, the stump's shape is tightly constrained — it sits near the c=1 end of this spectrum: low variance, because there just isn't enough flexibility in "one split" for sampling noise to meaningfully change the outcome, and high bias, because a single split almost never captures a genuinely complex decision boundary. An unpruned decision tree grown to full depth, by contrast, keeps splitting until every leaf is pure, which means it will happily carve out a leaf around a single mislabeled or noisy point if that's what's needed to drive training error to zero — it sits near the c=10 end: low bias, because with enough splits it can represent almost any decision boundary on average, and high variance, because exactly where those fine-grained splits land depends heavily on which specific noisy points happened to be in that particular training draw. Neither sits at the sweet spot on its own — which is precisely why bagging a forest of full-depth trees (attacking the stump's opposite failure mode from the variance side) is such a common fix, covered later.
Fix a single input point x. f(x) = E[Y|X=x] is the true regression function from the previous lesson — the Bayes-optimal target under squared loss. \hat f is the model actually fit on one particular, finite, randomly drawn training set. The expectations E[...] on the right-hand side are taken over the randomness of which training set got drawn — imagine refitting on thousands of fresh samples from the same distribution and averaging. σ² is the variance of the noise in Y around f(x) — it doesn't depend on the model at all.
Fix x and start from the quantity we actually care about — the expected squared error of the fitted model against a fresh, noisy observation Y at that point:
Write the noisy observation as the true function plus zero-mean noise, Y = f(x) + ε, with E[ε] = 0, Var[ε] = σ², and ε independent of the training set (and hence independent of \hat f). Substituting and regrouping:
Squaring and taking the expectation term by term:
The cross term vanishes: since ε is independent of \hat f (and f(x) is just a fixed constant), the expectation of the product factors into a product of expectations, and E[ε] = 0 kills it:
and the last term is exactly the noise variance, since E[ε] = 0 means E[ε²] = Var[ε] = σ². So we already have:
Now expand the remaining term. Let m = E[ˆf(x)] denote the average prediction over training sets, and add and subtract it:
The second piece, m − f(x), is just a fixed number once you know the model class and the true function — it does not vary as the training set varies, because m is itself already an average over all training sets. Squaring and taking the expectation (over training-set draws) term by term again:
The middle term vanishes too: E[f̂(x) − m] = E[f̂(x)] − m = m − m = 0 by the very definition of m. What's left is exactly the two named quantities:
Putting every piece back together gives the full decomposition:
Where this is used: this decomposition is the theoretical justification for two entire families of techniques covered in later modules. Regularization (ridge and lasso regression) deliberately shrinks a model toward simpler answers, which introduces some bias — but if it removes proportionally more variance, the net sum can go down, even though the fit is technically "more wrong" on average. Ensembling (bagging, random forests) works from the opposite end: averaging many independently fit, low-bias, high-variance models cancels out their disagreements without touching each individual model's bias, driving the variance term toward zero while leaving bias alone. Neither trick would make sense without first knowing that error genuinely splits into these three independent, additive pieces.
Watch the automatic sweep trace the U-shape once, then drag the slider yourself — bias² falls, variance rises, and their sum traces the classic U-shape with an interior minimum, not at either extreme.
Bias and variance are defined over an imaginary population of retrainings, but you can estimate them from real data with bootstrap resampling: draw many resampled training sets, fit the same model on each, and look at how the predictions at one fixed test point behave. All three implementations below run the identical experiment — 200 bootstrap-resampled training sets, a degree-3 polynomial fit, predictions collected at a single fixed x — and print the same three numbers so you can see the decomposition verified numerically, not just asserted algebraically.
import numpy as np
rng = np.random.default_rng(42)
# ---- 1. A known "true" function and a noisy data generator around it ----
def true_function(x):
return np.sin(1.5 * x) + 0.3 * x
SIGMA = 0.3 # true irreducible noise std-dev
N_TRAIN = 25 # points per training set
DEGREE = 3 # polynomial degree = "model complexity" for this experiment
X_TEST = 1.2 # fixed point at which we measure bias/variance
N_BOOTSTRAP = 200
def make_training_set(n):
x = rng.uniform(-3, 3, size=n)
y = true_function(x) + rng.normal(0, SIGMA, size=n)
return x, y
def fit_polynomial(x, y, degree):
# Build the design matrix by hand and solve the normal equations
# (X^T X) beta = X^T y directly -- no sklearn involved.
X = np.vstack([x ** p for p in range(degree + 1)]).T
beta = np.linalg.solve(X.T @ X, X.T @ y)
return beta
def predict(beta, xs):
degree = len(beta) - 1
X = np.vstack([np.asarray(xs) ** p for p in range(degree + 1)]).T
return X @ beta
# ---- 2. Bootstrap: fit on many resampled training sets, evaluate all at X_TEST ----
predictions = []
observed_targets = []
for _ in range(N_BOOTSTRAP):
x, y = make_training_set(N_TRAIN)
beta = fit_polynomial(x, y, DEGREE)
predictions.append(predict(beta, [X_TEST])[0])
observed_targets.append(true_function(X_TEST) + rng.normal(0, SIGMA))
predictions = np.array(predictions)
observed_targets = np.array(observed_targets)
# ---- 3. Decompose ----
f_true = true_function(X_TEST)
mean_prediction = predictions.mean()
bias_sq = (mean_prediction - f_true) ** 2
variance = predictions.var()
noise_floor = SIGMA ** 2
empirical_mse = np.mean((observed_targets - predictions) ** 2)
print(f"Bias^2 = {bias_sq:.4f}")
print(f"Variance = {variance:.4f}")
print(f"Irreducible noise = {noise_floor:.4f}")
print(f"Sum of the three = {bias_sq + variance + noise_floor:.4f}")
print(f"Observed MSE = {empirical_mse:.4f}")- A linear model on housing prices underfits whenever price genuinely depends on curved interactions (location premium that itself scales with square footage, diminishing returns on extra bedrooms) — no matter how much training data you throw at it, a straight line can't bend to capture that curvature. That's high bias: the average prediction is off, and more data doesn't fix it.
- A linear model fit to non-linear physics or chemistry data fails the same way for a more fundamental reason: reaction rates that follow an Arrhenius (exponential) temperature dependence, or drag forces that scale with velocity squared, are not approximately linear over any practically useful range. Fitting a straight line to such data doesn't just miss noise — it misses the actual shape of the physical law, so residuals show a clear systematic curve no matter how many more measurements you collect. This is textbook high bias: the mismatch is between the model's functional form and the truth, not between the model and any one dataset's sampling luck.
- An un-pruned decision tree grown to full depth on a small tabular dataset (a few hundred rows, say, in a churn or fraud model) partitions the training set until every leaf is pure — including leaves built around single noisy outliers. Retrain on a slightly different sample and the tree's splits, especially near the leaves, can look completely different, and its test accuracy can swing by several points between otherwise-equivalent train/test splits. That's high variance: the average fit may be fine, but any one fit is unreliable, and small datasets make it worse because there's less signal to drown out each split's sensitivity to individual points.
- Random forests and bagging (a later module) exist specifically to attack the variance term, not the bias term: average the predictions of many independently grown, high-variance, low-bias trees, and the disagreements between them cancel out while each tree's individual bias is untouched — variance drops sharply, bias barely moves. This is why bagging works so well on exactly the un-pruned-tree scenario above but does little for a model that's underfitting (a single decision stump, or the linear model on curved data) — you can't average away a systematic error that every copy in the ensemble shares.
- Ridge and lasso regression (a later module) attack the same problem from the other side: shrinking coefficients introduces a small, deliberate amount of bias, but if it removes a larger amount of variance, the net expected error goes down — the whole point of regularization is trading a little bias for a lot less variance.
- Early stopping when training a neural network is a direct bias-variance lever, even though it never touches the architecture. Training longer lets the weights drift further from their (high-bias, low-variance) random initialization toward a low-bias fit that has memorized sample-specific noise — high variance across different random seeds or slightly different training sets. Stopping training early, based on a held-out validation curve turning upward, is a way of dialing back effective model complexity without deleting a single parameter.
- Choosing k in k-nearest-neighbors is one of the most literal bias-variance dials in all of machine learning, because k directly controls how many training points get averaged into each prediction. k=1 predicts using the single closest training point — essentially zero bias (it can represent arbitrarily jagged decision boundaries) but very high variance (change one nearby training point and the prediction at that location can flip entirely). A large k averages over many neighbors, smoothing the boundary and stabilizing predictions across resamples (lower variance) at the cost of blurring over genuine local structure (higher bias) — k is quite literally the c-axis of this lesson's diagram, relabeled.
- Boosting a sequence of weak learners (covered in a later module) shows the opposite pairing works too: a single shallow decision stump is a deliberately high-bias, low-variance model that, on its own, underfits badly. Combining hundreds of such stumps — each one fit to correct the previous ensemble's remaining errors — can drive bias down close to zero while keeping variance under control, often beating a single large, low-bias, high-variance model outright. It's a reminder that "reduce variance by ensembling" (bagging) and "reduce bias by ensembling" (boosting) are two genuinely different mechanisms, aimed at opposite ends of this same decomposition.
- Treating "more complex model" as unconditionally worse or unconditionally better, instead of recognizing that where the sweet spot sits is entirely data- and problem-dependent — a model that's too simple for one dataset may be exactly right, or even still too simple, for another.
- Forgetting that the
σ²term is a hard floor: no model, however well chosen or well tuned, can push expected error below the irreducible noise in the data itself. If a "better model" claims near-zero error on genuinely noisy data, be suspicious of the evaluation, not impressed by the model. - Assuming the clean U-shape here is the whole story for every model class. Modern, heavily overparameterized models (very large neural networks in particular) can show a double descent pattern — error rises then falls a second time past the point where the model can perfectly fit the training data — which the simple picture in this lesson does not predict. The next lesson covers that phenomenon directly; for now, just flag it as a known exception to the classical U-shape, not a contradiction to resolve here.
Going deeper
This exact decomposition — clean, additive, three named terms — is a special property of squared-error loss. It comes directly from expanding a square, which is why the algebra above works out so neatly. Swap in a different loss (0-1 loss for a classifier, say) and there is no equally clean closed-form split into "bias" and "variance" terms that sum to the loss — classification error can still be usefully analyzed through bias-like and variance-like effects, but the tidy additive identity proved above genuinely does not carry over.
It's also worth not conflating this lesson's technical vocabulary with everyday usage. "Bias" here is a precise statistical property of an estimator — how its average prediction across hypothetical resampled training sets compares to the truth. That is a different concept from "bias" in the sense of a training set that systematically under-represents some group or scenario (a biased dataset, sometimes called sampling or selection bias). The two uses of the word are related in spirit — both mean "systematically off in some direction" — but they are not interchangeable, and papers that use one term while meaning the other cause real confusion.
You train a degree-1 (linear) model and a degree-15 (very wiggly) polynomial model on the same small dataset. The degree-15 model gets almost perfect training error but wildly different-looking fits every time you resample the training data slightly. Which term dominates its expected test error, and what does that predict about the linear model by contrast?
The degree-15 model is dominated by variance: it has enough capacity to bend to whatever specific noise is in the training sample, so its fit is close to unbiased on average but highly unstable from one training set to the next — exactly what 'wildly different-looking fits on resampling' describes. By contrast, the degree-1 model, being far too rigid to track sample-specific noise, will look nearly identical across resamples (low variance) but will systematically miss any real curvature in the true function (high bias). Neither model is 'better' in general — the decomposition just tells you which lever to pull: reduce the wiggly model's variance (regularize, get more data, or ensemble it), or reduce the linear model's bias (add capacity).
Every model's expected squared error at a point splits exactly into three additive pieces: bias² (systematic miss, from a model too rigid to represent the truth), variance (instability, from a model sensitive to which particular training sample it saw), and an irreducible noise floor no model can shrink. Model-complexity choices, regularization, and ensembling are three different practical answers to the same underlying question this decomposition poses: which of the two controllable terms, bias or variance, is worth trading against the other for your specific data.