Ordinal Regression / Classification
Cumulative-link models and the proportional-odds assumption.
On this page
Beginner: some categories have a natural order — a 1-to-5 star rating, a disease staged as poor/fair/good/excellent, a credit risk tier of low/medium/high. If you feed labels like these into the multi-class strategies from section 2.4.4 (one-vs-rest, one-vs-one) or treat them as plain softmax classes, the model has no idea "3 stars" sits between "2 stars" and "4 stars" — predicting 1 star instead of 5 is scored exactly as wrong as predicting 4 stars instead of 5. Ordinal regression is the fix: it builds the order directly into the model instead of throwing it away.
Intermediate: the trick is to imagine one continuous, unobserved ("latent") score for every example — literally the same linear score z = θᵀx that logistic regression computes in section 2.4.1 — and to imagine that score getting sliced into ordered bins by a set of threshold cut-points. An example falls into category k exactly when its latent score lands between cut-points τ_{k-1} and τₖ. Fitting the model means learning both the direction θ and the cut-points τ jointly from data.
Advanced: this is a cumulative-link model: instead of modeling each category's probability directly, it models the CUMULATIVE probability P(y ≤ k | x) as a link function applied to τₖ − θᵀx. Using the logistic sigmoid as that link gives the classic ORDERED LOGIT / proportional odds model — the workhorse of this lesson, and the natural place to end Module 4, which started this whole chapter with a single binary linear score.
One shared linear score θᵀx (section 2.4.1), K−1 ordered thresholds τ₁ < τ₂ < ⋯ < τ_{K-1} with τ₀ = −∞, τ_K = +∞, and the sigmoid σ as the link.
1. The latent-variable formulation. Assume every example has an unobserved continuous score z = θᵀx + ε, exactly the linear predictor from section 2.4.1, plus a noise term ε. The observed ordinal label is what you get by chopping the real line into K ordered intervals with cut-points τ₁ < τ₂ < ⋯ < τ_{K-1}:
A high latent score lands in a high category, a low one in a low category, and the thresholds mark where one category ends and the next begins — this is precisely the number-line picture in the first diagram below.
2. From latent cut-points to a cumulative-link model. Rather than modeling z itself (it's never observed), work with the CUMULATIVE event {y \le k}, which by the definition above is exactly the event {z \le \tau_k}:
If the noise ε follows a standard logistic distribution, its CDF is exactly the sigmoid σ, so this becomes the cumulative logit model:
(A Gaussian noise assumption instead gives ordered PROBIT, using Φ in place of σ — same construction, different link. This lesson works with the logit version throughout, matching logistic regression's link from 2.4.1.)
3. Individual category probabilities fall out as telescoping differences. Category k's own probability is the cumulative probability up to k minus the cumulative probability up to k−1:
Summed over all k = 1, …, K this telescopes to P(y ≤ K) − P(y ≤ 0) = 1 − 0 = 1, so the category probabilities are automatically a valid distribution. And here is the rigorous point: each individual term is guaranteed non-negative — not just in this construction, but necessarily — because τ_{k-1} < τₖ (the thresholds are ordered) and σ is strictly monotonically increasing, so σ(τₖ − θᵀx) ≥ σ(τ_{k-1} − θᵀx) for every x. Break the ordering — let some τₖ < τ_{k-1} — and this same algebra can produce a negative "probability," which is exactly why every fitting procedure for this model must enforce τ₁ < τ₂ < ⋯ < τ_{K-1} as a hard constraint, not a suggestion.
4. Why it's called "proportional odds." Rearranging the cumulative-link equation, the log-odds of the cumulative event {y \le k} is:
Notice which symbols carry a k and which don't: the intercept-like term τₖ depends on the threshold, but the SAME θ — same direction, same magnitude, for every predictor — appears for every k. Now compare two individuals x₁ and x₂ at the SAME threshold k. Their log-odds difference is:
so the odds RATIO between the two individuals for that cumulative event is:
Crucially, the right-hand side has no k in it at all — the same odds ratio applies whichever threshold k you evaluate it at, from y \le 1 all the way to y \le K-1. The odds are "proportional" across every cut-point in the ordinal scale — hence proportional odds. If this assumption fails in reality — if a feature's true effect is genuinely stronger at distinguishing low categories than high ones, i.e. it would need a different effective slope at different thresholds — a single shared θ can't represent that, and the model is misspecified. This is a real, testable assumption (score tests and Brant-style tests exist for exactly this), and it is sometimes violated in practice — not a formality to wave through.
5. Fitting by maximum likelihood. Given training pairs (xᵢ, yᵢ), the log-likelihood is a direct sum of the log category probabilities derived above:
There is no closed form, so — exactly as with plain logistic regression — this is maximized by numerical gradient-based optimization (section 2.2.2), jointly over θ and the thresholds τ. The ordering constraint from step 3 has to be respected throughout the optimization, which in practice is handled by reparameterizing: fit τ₁ freely and represent each subsequent threshold as the previous one plus a strictly positive gap, e.g. τₖ = τ_{k-1} + exp(δₖ) for unconstrained δₖ, so the ordering holds automatically for any values the optimizer explores.
Drag τ1, τ2, or τ3 along the latent-score axis. The three cumulative curves P(y≤k|x) = σ(τk − z) shift with their own threshold, and the readout shows each category's telescoped probability at a representative individual (z = 0). Dragging is clamped to keep τ1 < τ2 < τ3 -- the ordering the derivation shows is required for valid probabilities.
80 points from 4 true ordered categories along one feature x. Toggle between the ordinal model's regions (one shared θ and ordered τ -- always contiguous and correctly ordered by construction) and a naive one-vs-rest treatment (4 independently-fit classifiers with no shared ordering constraint, which can disagree about which class wins where).
#include <cmath>
#include <iostream>
#include <vector>
double sigmoid(double z) { return 1.0 / (1.0 + std::exp(-z)); }
// P(y = k | x) via telescoping cumulative differences, k is 1-indexed, taus.size() == K-1.
double categoryProb(int k, double score, const std::vector<double>& taus) {
int K = static_cast<int>(taus.size()) + 1;
double upper = (k == K) ? 1.0 : sigmoid(taus[k - 1] - score);
double lower = (k == 1) ? 0.0 : sigmoid(taus[k - 2] - score);
return upper - lower; // non-negative because taus is kept strictly increasing
}
double negLogLikelihood(double theta, const std::vector<double>& gaps, double tau1,
const std::vector<double>& x, const std::vector<int>& y) {
std::vector<double> taus = {tau1};
for (double g : gaps) taus.push_back(taus.back() + std::exp(g)); // ordering by construction
double nll = 0.0;
for (size_t i = 0; i < x.size(); ++i) {
double score = theta * x[i];
double p = categoryProb(y[i], score, taus);
nll -= std::log(std::max(p, 1e-12));
}
return nll;
}
int main() {
// Small hand-built dataset: 4 ordered categories driven by one feature.
std::vector<double> x = {-2.5, -2.0, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, 2.0, 2.5};
std::vector<int> y = {1, 1, 1, 2, 2, 3, 3, 3, 4, 4};
double theta = 0.0, tau1 = -1.0;
std::vector<double> gaps = {0.0, 0.0}; // log-gaps for tau2, tau3
double lr = 0.05;
for (int iter = 0; iter < 2000; ++iter) {
double eps = 1e-5;
double base = negLogLikelihood(theta, gaps, tau1, x, y);
double gTheta = (negLogLikelihood(theta + eps, gaps, tau1, x, y) - base) / eps;
double gTau1 = (negLogLikelihood(theta, gaps, tau1 + eps, x, y) - base) / eps;
std::vector<double> gGaps(gaps.size());
for (size_t j = 0; j < gaps.size(); ++j) {
auto g2 = gaps; g2[j] += eps;
gGaps[j] = (negLogLikelihood(theta, g2, tau1, x, y) - base) / eps;
}
theta -= lr * gTheta / x.size();
tau1 -= lr * gTau1 / x.size();
for (size_t j = 0; j < gaps.size(); ++j) gaps[j] -= lr * gGaps[j] / x.size();
}
std::cout << "theta=" << theta << " tau1=" << tau1
<< " tau2=" << tau1 + std::exp(gaps[0])
<< " tau3=" << tau1 + std::exp(gaps[0]) + std::exp(gaps[1]) << "\n";
return 0;
}- Customer satisfaction / survey ratings — "very dissatisfied" through "very satisfied," where the categories are ordered but the gaps between them aren't necessarily equal.
- Credit risk tiers — low / medium / high risk, where confusing adjacent tiers is a much smaller error than confusing the two extremes.
- Disease severity staging — mild / moderate / severe / critical, straight from a single latent "how sick is this patient" score.
- Product review star ratings — 1 to 5 stars, the canonical ordinal target, and a running example throughout this lesson.
- Education / grade levels — letter grades or proficiency bands (below basic / basic / proficient / advanced) assigned from an underlying continuous score.
- Treating ordinal labels as plain nominal multi-class (softmax, one-vs-rest, one-vs-one — section 2.4.4) — the order information is simply discarded, and "predicted 1 instead of 5" is scored identically to "predicted 4 instead of 5."
- Going the other direction and treating the integer-coded labels as a plain regression target — this silently assumes the categories are EQUALLY SPACED on some numeric scale (that the gap from "fair" to "good" equals the gap from "good" to "excellent"), an assumption that's rarely justified and that the cumulative-link model doesn't require at all — it lets the thresholds fall wherever the data puts them.
- Fitting the proportional-odds model and never checking the proportional-odds assumption itself. If a predictor's real effect differs meaningfully across thresholds, forcing one shared
θgives a misspecified model that can fit visibly worse than what a per-threshold slope would achieve — this is testable, not just theoretical.
Going deeper
When the proportional-odds assumption genuinely fails, the usual next step isn't to abandon the cumulative-link idea — it's to relax it. The generalized ordered logit (partial proportional odds) model lets a subset of predictors have their own threshold-specific slope θₖ instead of one shared θ, while keeping the rest of the predictors constrained to the proportional-odds assumption — a middle ground between full proportional odds and fitting K−1 completely independent binary models (which would throw away the shared structure and could reintroduce out-of-order regions like the naive one-vs-rest diagram above). It's also worth noticing the family resemblance to survival analysis's cumulative hazard framing, covered later in the Specialized Learning Problems module — both build a monotone family of cumulative probabilities from thresholds/times against a single linear score, just with different interpretations of what the "categories" are.
Why does the strict ordering τ1 < τ2 < ... < τ(K-1) guarantee that every category probability P(y=k|x) comes out non-negative?
Each category probability is a telescoping difference, P(y=k|x) = σ(τk − θᵀx) − σ(τ(k-1) − θᵀx). The sigmoid σ is strictly increasing, so σ(a) ≥ σ(b) whenever a ≥ b. Because the thresholds are ordered, τk > τ(k-1), which makes τk − θᵀx > τ(k-1) − θᵀx for the SAME x — and pushing that through the monotone σ preserves the inequality, giving σ(τk − θᵀx) ≥ σ(τ(k-1) − θᵀx), i.e. a non-negative difference. Break the ordering and this guarantee breaks with it — a fit with τk < τ(k-1) can produce a negative 'probability,' which is exactly why every practical fitting procedure enforces the ordering as a hard constraint (e.g. via positive reparameterized gaps) rather than trusting the optimizer to respect it on its own.
This closes out Module 4's arc: 2.4.1 built a single linear score into a binary decision via the sigmoid; 2.4.2/2.4.4 extended that to K unordered classes; 2.4.5 extended it again to multiple simultaneous labels; and this lesson has shown the fourth and final shape the same linear score can take — K ORDERED classes, produced by slicing one latent score with ordered thresholds under the proportional-odds assumption. Module 5, Generative & Instance-Based Classifiers, now approaches classification from a genuinely different angle: instead of modeling P(y|x) directly from a linear score as every method in this module has, it starts from P(x|y) and reaches a classification decision via Bayes' rule.