Linear Classification
The same linear score as regression, squashed into a class decision.
On this page
Module 3 was entirely about predicting a continuous number. This module asks the other half of supervised learning's central question: what changes when the target is a category instead — spam or not spam, which of ten digits, which of several possible diagnoses? The surprising answer is: less than you'd expect. Every method here still computes a linear combination θᵀx of the input features — exactly the Xθ from OLS (section 2.3.1) — and the entire subject of this module is what happens to that same linear score once it's forced through a function that turns it into a class decision instead of a real-valued prediction.
That's also why logistic regression already made a cameo appearance in section 2.3.9 (Generalized Linear Models), as the canonical logit-link Bernoulli GLM fit by IRLS. This module doesn't repeat that IRLS derivation — instead it builds logistic regression up from its own first principles (the sigmoid, log-odds, and cross-entropy loss), derives its gradient and Hessian directly, and then spends the rest of the module on everything a GLM framing doesn't cover: the mistake-driven Perceptron, the softmax generalization to many classes at once, and the practical machinery (one-vs-rest, ECOC, classifier chains, ordinal thresholds) needed to turn a binary classifier into one that handles multi-class, multi-label, and ordered-category problems.
Every model in this module starts from the same linear score and only changes what turns that score into a decision:
Logistic regression sets g to the sigmoid and fits θ by minimizing cross-entropy — a smooth, probabilistic loss with a unique minimum (section 2.2.1's convexity, revisited here via the Hessian). The Perceptron sets g to the sign function and fits θ by a mistake-driven update with no loss surface to speak of at all — a genuinely different, older, and more primitive learning rule that this module derives and contrasts directly against logistic regression's gradient descent. Softmax regression is the many-class generalization of the same score-then-squash idea, replacing one sigmoid with K competing linear scores normalized into a probability simplex. And once a binary or probabilistic classifier exists, the module's final three lessons are about composition: how to build a multi-class, multi-label, or ordinal classifier out of it without inventing a new model from scratch.
- 2.4.1 Logistic Regression — the sigmoid, log-odds, binary cross-entropy loss, its gradient derived by hand, and the Hessian that proves the loss is convex.
- 2.4.2 Softmax / Multinomial Regression — the direct generalization to
K > 2classes: the softmax function, categorical cross-entropy, and the gradient with respect to every class's logits at once. - 2.4.3 Perceptron — the original linear classifier: a mistake-driven update rule, the margin, and the Novikoff theorem guaranteeing convergence on linearly separable data in finitely many steps.
- 2.4.4 Multi-class Strategies — one-vs-rest, one-vs-one, and error-correcting output codes: turning any binary classifier into a multi-class one without changing the base model at all.
- 2.4.5 Multi-label Classification — binary relevance, classifier chains, and label powerset, for problems where each example can carry more than one label at once.
- 2.4.6 Ordinal Regression / Classification — cumulative-link models and the proportional-odds assumption, for categories with a natural order (poor / fair / good / excellent) that plain multi-class methods throw away.
- Treating "classification" as a single problem type — binary, multi-class, multi-label, and ordinal targets each throw away different information if you use the wrong strategy (e.g. one-vs-rest on an ordinal target discards the ordering entirely).
- Forgetting that a linear classifier's decision boundary is linear in the features, not necessarily in the original input space — exactly as in polynomial/basis regression (section 2.3.6), a basis expansion turns straight-line boundaries into curved ones without leaving the linear-classification framework.
Going deeper
Logistic regression and the Perceptron look superficially similar — both are sign-or-sigmoid of a linear score — but they come from different intellectual traditions and behave differently in exactly the cases this module highlights: the Perceptron has no notion of probability and can cycle forever on non-separable data, while logistic regression's convex, smooth loss always converges to a well-defined (if not perfectly separating) solution. Seeing both derivations side by side is the fastest way to understand why the field mostly moved from the Perceptron to probabilistic, gradient-based classifiers — and why the Perceptron is still the conceptual ancestor of every neural network's single unit.
Start with 2.4.1, Logistic Regression — its sigmoid, cross-entropy loss, and gradient derivation are the template that softmax regression (2.4.2) directly generalizes, and the probabilistic output it produces is exactly what the multi-class and multi-label composition strategies later in this module assume they have available.