Maximum Likelihood & MAP
The single inference principle almost every loss function in ML derives from.
On this page
Beginner: imagine you find a coin that might be biased and flip it a handful of times. Likelihood just asks: "if the coin's true bias were p, how probable would it be to see exactly the heads/tails sequence I actually observed?" Maximum likelihood estimation (MLE) tries every candidate value of p and picks the one that makes your actual data the most probable outcome. If you flip 10 times and get 7 heads, the value of p that makes "7 heads out of 10" the single most likely result is exactly p = 0.7 — which probably matches your intuition already, and that's not a coincidence.
Intermediate: in practice you never work with the raw probability of the data directly — you work with its logarithm, the log-likelihood. Three reasons, all practical: turning a product of many small probabilities into a sum is easier to differentiate term by term; a product of, say, 10,000 probabilities each around 0.1 underflows to numerically indistinguishable-from-zero on any computer, while the sum of 10,000 log-probabilities stays a perfectly ordinary-sized number (the same underflow concern behind "logsumexp"-style numerical stability tricks elsewhere in ML); and because log is a strictly increasing function, whatever value of p maximizes the log-likelihood also maximizes the likelihood itself — nothing about where the maximum sits is lost by taking the log.
Advanced: MLE treats the parameter as a completely blank slate — it uses only the data in front of it and nothing else. MAP (maximum a posteriori) estimation adds one ingredient: a prior, a probability distribution encoding what you believed about the parameter before seeing any data, and then picks the parameter value that maximizes log-likelihood plus log-prior together. The prior acts like a gentle thumb on the scale, and the size of its effect is not fixed — as you collect more and more data, the log-likelihood term grows in magnitude (more data points being multiplied in) while the log-prior term stays exactly the same size, so its relative influence shrinks toward nothing and MAP converges to whatever MLE would have given anyway. With very little data, though, the prior dominates and quietly does the job of regularizing the estimate away from the wild, noise-driven extremes a tiny sample can produce.
Worked example — an A/B test click-through rate. A new checkout-button design is shown to 8 visitors, and 6 of them click through. That is the coin-flip problem again with new labels — a click is "heads," a non-click is "tails" — so k = 6, n = 8, and MLE reports p̂ = 6/8 = 0.75: a flat 75% click-through rate, full stop, based on eight visitors. Now suppose company-wide history says checkout buttons on pages like this one click through around 50% of the time — encode that as a mild Beta(3, 3) prior, whose mode is exactly 0.5. Plugging into the MAP formula derived below, p̂ = (k+α-1)/(n+α+β-2) = (6+3-1)/(8+3+3-2) = 8/12 ≈ 0.667. MAP has pulled the raw 75% estimate back toward the historical 50% baseline — not all the way, but noticeably — because eight visitors isn't much evidence to override what history suggests. There's a useful way to read that α − 1 and β − 1 in the formula: they act as pseudo-observations the prior silently contributes before any real data arrives. A Beta(3, 3) prior is worth exactly 2 pseudo-clicks and 2 pseudo-non-clicks, no more — which is why it takes only a moderate amount of genuine traffic (a few dozen more visitors, say) to swamp it and pull the estimate back toward the raw 75% the data alone is pointing at.
MLE maximizes log-likelihood alone; MAP maximizes log-likelihood plus the log of a prior belief P(θ) over the parameter itself. Drop the prior term entirely and MAP is MLE — MLE is just MAP under a flat (uniform, "no opinion") prior.
Part 1 — the Bernoulli MLE. Flip a coin n times independently and observe k heads. The probability of that exact sequence, as a function of the candidate bias p, is the likelihood:
Take the log to turn the product implicit in that exponentiation into a sum:
Differentiate with respect to p and set the derivative to zero:
Cross-multiply and solve directly:
— exactly the observed heads fraction, confirming the beginner intuition above with real calculus.
Part 2 — the Beta-prior MAP. Now add a belief about p before seeing data: a Beta(α, β) prior, whose density has the proportional form:
Multiplying this prior by the likelihood above gives the posterior, and because a Beta prior combined with a Bernoulli/Binomial likelihood always produces another Beta distribution — the two are conjugate — the algebra collapses cleanly without needing the general theory of why that conjugacy holds:
Take the log, differentiate, and set to zero exactly as before (the missing normalizing constant drops out under differentiation, since it has no p in it):
Two sanity checks confirm this matches the plain-English story exactly. As n → ∞ with α, β fixed, divide numerator and denominator by n:
— the prior's influence washes out and MAP collapses onto MLE, exactly as claimed above. At the opposite extreme, with zero data (n = 0, k = 0):
which is precisely the mode of the Beta(α, β) prior itself — with no data at all, MAP simply returns your prior belief unchanged, since there's nothing yet to update it with.
Part 3 — squared loss is secretly Gaussian MLE. This is the identity that justifies calling MLE/MAP the unifying idea of the whole module. Assume a linear-Gaussian model, exactly the setup behind ordinary linear regression:
Equivalently, y_i given x_i and w is Gaussian with mean w^Tx_i, so its density is:
Assuming the observations are conditionally independent given w, sum the log of this density over all n data points:
The first term doesn't contain w at all — it's a constant as far as the maximization is concerned. Only the second term depends on w, so:
Maximizing the Gaussian log-likelihood is algebraically identical to minimizing sum-of-squared-error — precisely the ordinary least squares (OLS) objective, and precisely the squared-loss result that Statistical Decision Theory (2.1.2) derives from a completely different starting point (choosing the Bayes-optimal predictor under squared loss). These are not two coincidentally similar ideas; they are the same optimization problem, arrived at from two directions.
The same trick extends to MAP. Give the weights a Gaussian prior, w ~ N(0, τ²I), whose log-density is −‖w‖²/(2τ²) plus a constant. Adding that to the log-likelihood above and flipping the sign to turn "maximize" into "minimize":
— squared error plus an L2 penalty proportional to ‖w‖², which is exactly ridge regression's objective, with the regularization strength pinned to the concrete ratio σ²/τ². Ridge regression is MAP estimation under a Gaussian prior on the weights — not "analogous to," literally the same optimization problem. Swap in a Laplace prior instead of a Gaussian one and the same style of derivation produces an L1 penalty instead of L2 — exactly the Lasso objective. Both regressions, and the full bias/variance mechanics of why they help, are developed in detail in a later Regression module; the point here is just that they're falling out of this one principle, not a new one.
Where this is used: this is the single biggest unifying idea in the whole Foundations module. Essentially every "loss function" this chapter introduces later is the negative log-likelihood of some assumed noise or data-generating distribution in disguise — squared error is negative-log-Gaussian noise, cross-entropy/log-loss for classification is the negative log-likelihood of a Bernoulli/categorical label, and even hinge loss's margin-maximizing behavior traces back to related generative assumptions. Symmetrically, every regularization penalty is the negative log of some prior over the parameters — L2 is a Gaussian prior, L1 is a Laplace prior. "Choose a loss function" and "choose a likelihood," "add a regularizer" and "choose a prior," are the same design decision wearing two different names.
Dashed orange is the prior density, solid blue the posterior. The green dashed line marks the MLE (pure data, k/n); the violet line marks the MAP (peak of the posterior). Push heads+tails up with the prior fixed and watch MAP slide onto MLE.
Part 1 of each tab reproduces the closed-form Bernoulli MLE/MAP derived above; part 2 proves the loss ⟺ likelihood correspondence in code, not just algebra — the OLS normal- equations solution and a direct numerical maximizer of the Gaussian log-likelihood land on the same answer, and swapping LinearRegression for Ridge is exactly swapping a flat prior on w for a Gaussian one.
import random
import math
def bernoulli_mle(heads: int, tails: int) -> float:
n = heads + tails
if n == 0:
return 0.5 # undefined with no data at all; fall back to "no opinion"
return heads / n
def bernoulli_map(heads: int, tails: int, alpha: float, beta: float) -> float:
n = heads + tails
denom = n + alpha + beta - 2
if abs(denom) < 1e-12:
return 0.5
return (heads + alpha - 1) / denom
heads, tails = 2, 8
alpha, beta = 3.0, 3.0
print("MLE p_hat =", bernoulli_mle(heads, tails))
print("MAP p_hat =", bernoulli_map(heads, tails, alpha, beta))
# ---- Part 2: OLS via the normal equations reproduces the Gaussian-MLE optimum ----
def transpose(A):
return [list(row) for row in zip(*A)]
def matmul(A, B):
Bt = transpose(B)
return [[sum(a * b for a, b in zip(row, col)) for col in Bt] for row in A]
def mat_vec(A, v):
return [sum(a * x for a, x in zip(row, v)) for row in A]
def solve_linear_system(A, b):
# Plain Gaussian elimination with partial pivoting -- no numpy involved.
n = len(A)
M = [row[:] + [b[i]] for i, row in enumerate(A)]
for col in range(n):
pivot = max(range(col, n), key=lambda r: abs(M[r][col]))
M[col], M[pivot] = M[pivot], M[col]
for r in range(col + 1, n):
factor = M[r][col] / M[col][col]
for c in range(col, n + 1):
M[r][c] -= factor * M[col][c]
x = [0.0] * n
for row in range(n - 1, -1, -1):
s = M[row][n] - sum(M[row][c] * x[c] for c in range(row + 1, n))
x[row] = s / M[row][row]
return x
random.seed(0)
true_w = [2.0, -1.5]
xs = [[1.0, random.uniform(-3, 3)] for _ in range(40)] # column 0 is the intercept term
ys = [true_w[0] * x[0] + true_w[1] * x[1] + random.gauss(0, 0.5) for x in xs]
Xt = transpose(xs)
XtX = matmul(Xt, xs)
Xty = mat_vec(Xt, ys)
w_ols = solve_linear_system(XtX, Xty)
print("OLS via normal equations:", w_ols)
def neg_log_likelihood(w, sigma2=0.25):
sse = sum((y - (w[0] * x[0] + w[1] * x[1])) ** 2 for x, y in zip(xs, ys))
return sse / (2 * sigma2) # dropping the additive constant -- it never affects the argmin
def grid_search_mle(step=0.02, span=25):
best_w, best_nll = None, math.inf
for i in range(-span, span + 1):
for j in range(-span, span + 1):
w = [w_ols[0] + i * step, w_ols[1] + j * step]
nll = neg_log_likelihood(w)
if nll < best_nll:
best_nll, best_w = nll, w
return best_w
w_mle_search = grid_search_mle()
print("Grid-search maximizer of the Gaussian log-likelihood:", w_mle_search)
print("Same answer either way -- minimizing squared error IS maximizing Gaussian likelihood.")- A/B testing and click-through-rate estimation — with only a handful of impressions on a new ad variant, plain MLE can report a wildly overconfident rate (a single click out of three impressions gives an MLE CTR of 33%). MAP/Bayesian smoothing with a sensible prior tempers exactly this kind of small-sample overconfidence, the same Beta-Bernoulli mechanics derived above, just relabeled as clicks and impressions.
- Ridge regression in practice is, as derived above, nothing more than MAP estimation of linear weights under a Gaussian prior — the "regularization strength" hyperparameter is literally a noise-to-prior variance ratio. The full treatment of why and when this helps, alongside Lasso, is developed in a later Regression module.
- Naive Bayes classifiers, covered in full in a later module, are a direct, explicit application of MLE — they estimate the parameters of an assumed generative distribution for each class (e.g. word frequencies) purely by maximum likelihood on labeled training data, then classify new points by whichever class distribution makes the observed features most likely.
- Language models estimating the probability of the next word/token given context are, at their statistical core, (heavily regularized) maximum likelihood estimates over enormous text corpora — every "next-token probability" a language model outputs is shaped by the same likelihood-maximization principle as the two-flip coin example above, just at a vastly larger scale and with the prior/regularization baked into the model architecture and training procedure rather than a hand-written Beta prior.
- Spam filters built on naive Bayes hit a sharp failure mode without a fix: if the word "viagra" never once appeared in the training set's ham folder, its MLE estimate of
P(word | ham) = 0exactly, and Bayes' rule then multiplies that zero straight through the whole calculation — any email containing the word gets classified as spam with total certainty, regardless of every other word in it, forever. The standard fix, Laplace (additive) smoothing — adding a small pseudo-count to every word before dividing — is exactly a MAP estimate under a uniform Dirichlet prior over the vocabulary, the multi-outcome generalization of the "pseudo-observations" reading of the Beta prior in the worked example above. - Recommender systems that factor a user-item ratings matrix into latent user and item vectors almost always add an L2 penalty on those vectors before fitting — unregularized factorization overfits badly to users and items with only a handful of ratings, memorizing noise instead of latent taste. By exactly the same ridge-regression identity derived above, that penalty is a Gaussian MAP prior over the latent factors; the algorithm is quietly doing Bayesian regularization even though most descriptions of it never use the word "prior."
- Trusting plain MLE with tiny samples: flip a coin twice, get two heads, and MLE reports
p̂ = 1— "this coin always lands heads," from two flips. Nothing in plain MLE stops it from being this confidently wrong; some form of prior or regularization is what tempers it (with a Beta(3,3) prior, the same two flips give MAPp̂ = 4/6 ≈ 0.667instead — still leaning toward heads, but not absurdly so). - Treating the choice of a prior as "cheating" or purely subjective bias sneaking into an otherwise objective calculation. Under the loss-correspondence view derived above, a prior is just a regularization penalty by another name, with well-studied statistical behavior (bias added, variance reduced) — no more "subjective" than choosing to add an L2 penalty, which almost nobody objects to on principle.
- Assuming MAP is unconditionally "better" than MLE. With reasonable amounts of data and a sensibly chosen prior, MAP rarely hurts and often helps. But a badly mismatched or overly strong prior, combined with little data to outweigh it, can bias an estimate just as badly as having no regularization at all lets an estimate overfit — regularization is a trade-off to be tuned, not a free win.
Going deeper
MAP is a genuinely useful, cheap approximation to full Bayesian inference — but it is not the same thing as it. Full Bayesian inference, covered in a later Probabilistic & Bayesian Methods module, never collapses the posterior down to a single point; it keeps the entire posterior distribution over the parameter, which lets you quantify uncertainty in downstream predictions (e.g. "here's a range of plausible click- through rates, and how confident we are in each") rather than reporting one number as if it were exact. MAP is the mode of exactly that same posterior — useful, computationally cheap, but strictly less informative than the whole distribution.
A subtlety worth flagging: MLE is invariant under reparameterization — if you estimate a transformed parameter (say, log p instead of p) by maximum likelihood, you get exactly the transform of the original MLE. Naive MAP does not share this property in general — the mode of a transformed density is not, in general, the transform of the original mode, because probability densities pick up a Jacobian (change-of-variables) factor under reparameterization that shifts where the peak sits. This is a genuine wrinkle for anyone reporting MAP estimates across different parameterizations of the same problem, and it's part of why some practitioners prefer reporting a posterior mean or the full posterior instead of a bare MAP point.
You flip a coin twice and observe 2 heads. Under a Beta(3, 3) prior, what are the MLE and MAP estimates of the bias p, and why do they differ so much?
MLE = k/n = 2/2 = 1 — with only two data points, MLE reports total certainty the coin always lands heads. MAP = (k+α-1)/(n+α+β-2) = (2+3-1)/(2+3+3-2) = 4/6 ≈ 0.667 — the Beta(3,3) prior, whose mode is 0.5, pulls the estimate back from that extreme. The gap between them is exactly the effect of having almost no data: with n this small, the prior term in the MAP objective is comparable in size to the likelihood term, so it has real influence. Flip the coin 200 times and get 140 heads instead, and MLE (0.7) and MAP under the same prior ((140+2)/(200+4) ≈ 0.696) become nearly indistinguishable — the prior's influence has washed out.
This closes out Foundations of Learning. The module opened with empirical risk minimization — pick the function, from an allowed hypothesis class, that minimizes average loss on the training data — and every topic since has been filling in a piece of what that means and why it works: which paradigm supplies the data (2.1.1), where the loss function itself comes from (2.1.2), why no single model can drive bias, variance, and irreducible error to zero simultaneously (2.1.3), how that trade-off shows up as measurable over/under- fitting as capacity changes (2.1.4), and what theoretical guarantees say about whether training performance transfers to new data at all (2.1.5). This lesson closes the loop: maximum likelihood and MAP estimation show that choosing a loss function is choosing a noise/data-generating distribution, and choosing a regularizer is choosing a prior — the "loss" and "regularization" vocabulary used everywhere else in this chapter and the "likelihood" and "prior" vocabulary used here are two names for the same underlying choice. Every later module in this chapter, whenever it introduces a new model or loss function, will point back to whichever of these six foundational ideas explains why that model is built the way it is — it's worth remembering which of the six is doing the explaining each time.