Non-negative Matrix Factorization
SVD's interpretable cousin — the technique behind topic models and eigenfaces' rival.
On this page
Beginner: SVD (section 1.10) factors a matrix into pieces that can contain negative numbers, which often makes the individual factors hard to interpret — a "negative amount of a topic" doesn't mean anything intuitive. Non-negative Matrix Factorization (NMF) factors a non-negative matrix into two smaller non-negative matrices instead, so every piece stays interpretable as "an amount of something."
Intermediate: because everything stays non-negative, NMF factors tend to be naturally sparse and additive rather than involving cancellation between positive and negative terms — this is why NMF on a document-term matrix tends to produce factors that read like genuine, human-interpretable "topics," each defined by a handful of strongly-weighted words, rather than an abstract rotated coordinate system.
Advanced: unlike SVD, NMF has no closed-form solution — it's solved iteratively (typically via multiplicative update rules or alternating least squares), the factorization isn't unique, and the optimization problem is non-convex (section 1.16), so different runs (or different random initializations) can converge to different, equally valid answers.
For an m×n matrix A and a chosen rank k, W is m×k and H is k×n — same shape pattern as a truncated SVD, but with the added non-negativity constraint that changes everything about how it must be solved.
Minimizing ‖A − WH‖²_F by ordinary gradient descent (section 1.17) on W would use W ← W − η·∇_W, where the gradient is ∇_W = 2(WH − A)Hᵀ. Nothing in that update prevents entries of W from going negative — a generic step size can easily overshoot past zero.
Lee and Seung's trick is to choose the step size η itself, separately for every entry, so it exactly cancels the negative part of the gradient. Split the gradient into its two non-negative pieces, ∇_W = 2(WHHᵀ) − 2(AHᵀ), and set the per-entry step size to η_{ij} = W_{ij} / (2(WHH^T)_{ij}) — half the reciprocal of the positive piece of the gradient, so that piece exactly cancels:
Every quantity on the right — W, A, H — is non-negative by assumption, and a ratio of non-negative numbers is non-negative. So W can shrink toward zero but can never cross it, and the same construction applied to H gives an equally safe multiplicative update. This is exactly why NMF solvers use this specific, seemingly ad-hoc update rule instead of plain gradient descent — it's the one choice of step size that makes non-negativity automatic rather than something that needs a separate projection or constraint step.
Where this is used: this is literally the default solver inside sklearn.decomposition.NMF (solver='mu'), and the same "adaptive step size cancels the negative part of the gradient" trick reappears in other constrained-optimization corners of ML wherever a variable must be kept non-negative throughout training.
A tiny document-term matrix (rows = documents, columns = words, entries = word counts) with two clear underlying topics — "sports" and "cooking" — factors via NMF into a W matrix whose columns represent "how much of each topic is in this document" and an H matrix whose rows represent "how strongly each word belongs to that topic." Every entry stays non-negative and directly interpretable: document 1 might be [0.9, 0.1] (mostly sports), and the "sports" row of H might have its largest weights on words like ball, score, and team. SVD on the exact same data would produce mathematically valid but far less interpretable factors, mixing positive and negative weights across unrelated words.
init='nndsvd' uses a non-negative-adapted SVD to pick a good starting point — a direct, practical bridge between sections 1.10 and this one.
import numpy as np
from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer
docs = [
"the team scored a goal in the match",
"the recipe needs flour sugar and eggs",
"the striker scored twice in the football match",
"bake the cake with sugar and flour",
]
X = TfidfVectorizer().fit_transform(docs)
model = NMF(n_components=2, init='nndsvd', random_state=0)
W = model.fit_transform(X) # document-topic weights, all >= 0
H = model.components_ # topic-word weights, all >= 0
print(W.round(2)) # each row sums up "how much of each topic" per document- Topic modeling — NMF on a term-document matrix is a fast, simple alternative to Latent Dirichlet Allocation for extracting human-readable topics.
- Image feature extraction — NMF on a matrix of face images famously extracts interpretable "parts" (eyes, noses, mouths) rather than the global, harder-to-interpret "eigenfaces" that PCA/SVD produce on the same data.
- Audio source separation — spectrograms are naturally non-negative (they're magnitudes), making NMF a natural fit for separating overlapping sound sources.
- Expecting NMF to reproduce the same factors every run — because the problem is non-convex and solved iteratively, different initializations genuinely converge to different (still valid) answers; always fix a random seed for reproducibility.
- Applying NMF to data containing negative values — it fundamentally requires a non-negative input matrix; this rules it out for raw embeddings or centered data without a preprocessing step.
Going deeper
NMF's lack of a unique solution is actually a meaningful mathematical fact, not just a numerical inconvenience — the factorization is only unique up to certain additional constraints (like sparsity or specific normalization schemes), an active research question for exactly which extra assumptions guarantee a unique, "correct" answer.
At the master level: NMF is a special case of the broader family of constrained matrix factorizations, which includes techniques enforcing sparsity, smoothness, or other structural priors on W and H — the same core "factor into two smaller matrices" idea from SVD and NMF, generalized further by swapping in whatever constraint best matches the structure you know your data actually has.