Locally Weighted Regression (LOESS)
Fitting a new tiny regression at every point, weighted by a sliding kernel.
On this page
Beginner: every earlier lesson in this module fits ONE global model to the whole dataset — one line, one penalized line, one set of spline coefficients. LOESS does something completely different: to predict at any point, it fits a BRAND NEW tiny regression using only the nearby training points, weighted by how close they are. Slide along the x-axis, refit at every spot, and the predictions trace out a smooth curve with no single global formula behind it at all.
Intermediate: "close" is defined by a KERNEL function and a BANDWIDTH — points right next to the query point get weight close to 1; points far away get weight close to 0. A small bandwidth means a very local, narrow window; a large bandwidth means a wide, nearly-global window.
Advanced: LOESS is fully non-parametric — it never commits to a global functional shape (line, polynomial, spline basis) at all. That flexibility is bought with a real cost: every single prediction requires solving a fresh weighted least-squares problem, and the bandwidth choice reintroduces exactly the bias-variance trade-off from Module 1 in a new form.
A separate weighted regression solved at every query point x₀, with diagonal weight matrix W(x₀) built from a kernel K and bandwidth h.
Generalize plain OLS's derivation (section 2.3.1) by weighting each squared residual: minimize Σᵢ wᵢ(x₀)(yᵢ−xᵢᵗθ)² where wᵢ(x₀) depends on how far xᵢ is from the query point x₀. Differentiate with respect to θ and set the gradient to zero — the algebra is identical to plain OLS's derivation with every term carrying an extra weight factor — giving exactly:
Plain OLS is the special case where every wᵢ(x₀)=1 regardless of x₀ — no locality at all, hence one single global fit rather than a different one at every query point.
Now the bandwidth trade-off, precisely: a very small h means only a handful of very nearby points get meaningful weight at each query — LOW BIAS (the local fit tracks genuinely local structure faithfully) but HIGH VARIANCE (each local fit rests on very little effective data, so it's noisy and jumps around with small changes in the sample). A very large h gives nearly every point equal weight everywhere — LOW VARIANCE (each local fit uses almost the whole dataset, very stable) but HIGH BIAS (it can't track any genuinely local curvature, and as h→∞ it degenerates into one single global linear fit — plain OLS again).
Where this is used: this bandwidth-driven bias-variance trade-off is exactly why choosing h well (typically via cross-validation) matters as much for LOESS as choosing λ does for ridge/lasso earlier in this module.
The shaded window shows which points currently matter; the red line is the tiny local weighted fit at the cursor's position. As the cursor sweeps left to right, each local prediction becomes one point on the smooth green LOESS curve.
The same scatter, kernel, and bandwidth, fitting two different local polynomials at every x0: a flat local weighted average (Nadaraya-Watson, dashed amber) versus a local weighted line (LOESS, solid green). They track each other closely in the interior, but drag the bandwidth up and watch them pull apart near the edges of the range, where only the local line can follow the data's slope.
#include <cmath>
#include <iostream>
#include <vector>
double gaussianKernel(double d, double h) {
return std::exp(-0.5 * (d / h) * (d / h));
}
double loessPredict(const std::vector<double>& x, const std::vector<double>& y, double x0, double h) {
double sw = 0, swx = 0, swy = 0, swxx = 0, swxy = 0;
for (size_t i = 0; i < x.size(); ++i) {
double w = gaussianKernel(x[i] - x0, h);
sw += w; swx += w * x[i]; swy += w * y[i];
swxx += w * x[i] * x[i]; swxy += w * x[i] * y[i];
}
double denom = sw * swxx - swx * swx;
if (std::abs(denom) < 1e-9) return swy / sw;
double slope = (sw * swxy - swx * swy) / denom;
double intercept = (swy - slope * swx) / sw;
return intercept + slope * x0;
}
int main() {
std::vector<double> x, y;
for (int i = 0; i < 60; ++i) {
double xi = -3.0 + 6.0 * i / 59;
x.push_back(xi);
y.push_back(std::sin(1.4 * xi) + 0.15 * xi);
}
for (double x0 : {-2.0, -1.0, 0.0, 1.0, 2.0}) {
std::cout << "x0=" << x0 << " loess=" << loessPredict(x, y, x0, 0.5) << "\n";
}
return 0;
}- Exploratory data analysis trend lines — the "LOESS smoother" routinely overlaid on scatter plots in statistical graphics (e.g.
geom_smoothin R's ggplot2). - Time-series trend extraction without assuming any specific parametric shape for the trend.
- Any dataset with a genuinely unknown, possibly complex relationship, where committing to a specific basis (polynomial degree, spline knots) feels premature.
- Local calibration curves in measurement and instrumentation, where different regions of the input range may need different local corrections.
- LOESS is computationally expensive at PREDICTION time — every new query point needs a fresh weighted least-squares solve, unlike a fitted linear/spline model that's solved once and evaluated cheaply forever after.
- Using a constant bandwidth when data density varies a lot across the range — sparse regions get too few effectively-weighted points at a fixed bandwidth, which is why some implementations define the window by "k nearest neighbors" instead of a fixed distance, adapting automatically to local density.
- Extrapolating outside the training range, where there are no nearby points to weight at all — the fit becomes undefined or wildly unreliable right at the edges.
Going deeper
LOESS sits on a spectrum of "local polynomial regression": the Nadaraya-Watson estimator is the simpler special case that fits a local WEIGHTED AVERAGE — a degree-0 local "polynomial" — at each query point, rather than LOESS's full local linear (or sometimes local quadratic) fit. Naming this precisely locates LOESS as the degree-1-or-higher generalization of that simpler, even more classical baseline.
As the bandwidth h grows very large, what does the LOESS fit converge to, and why?
Plain OLS — a single global linear fit. As h grows, the Gaussian (or any bounded) kernel's weights become nearly equal for every training point regardless of distance from the query point, so the weighted normal equations X^T W(x0) X θ = X^T W(x0) y converge to the unweighted normal equations from section 2.3.1. At that point every query point's local fit is solving the same weighted (nearly unweighted) problem, so the 'local' line becomes one single global line.
LOESS trades away every notion of a single global formula in exchange for maximum local flexibility, governed by nothing but a kernel and a bandwidth. The next lesson closes out this module with the opposite kind of flexibility trade: no functional form assumption at all, replaced by a single hard ORDER constraint — isotonic regression.