Triangular Jacobians & Normalizing Flows
Why flow-based generative models are designed the exact way they are.
On this page
Beginner: section 1.12 mentioned that transforming a probability distribution into new coordinates requires dividing by the absolute value of a Jacobian determinant, to keep total probability equal to 1. Normalizing flows, a family of generative models, are built entirely around this one fact — they model a complex distribution as a sequence of simple, invertible transformations applied to simple noise, tracking exactly how probability density changes at every step.
Intermediate: the catch is that computing a determinant (section 1.12) is normally expensive — O(n³) in general. Normalizing flows sidestep this entirely by deliberately designing every transformation so its Jacobian matrix is exactly triangular — and the determinant of a triangular matrix (section 1.13) is just the product of its diagonal, computable in O(n) time instead.
Advanced: this is a genuine architectural constraint, not an incidental detail — layers like RealNVP and MAF are specifically engineered so that each output dimension only ever depends on a strict subset of the input dimensions, which is precisely what forces the Jacobian into triangular form and makes the whole log-likelihood computation for training tractable at all.
The change-of-variables formula from probability, combined with the triangular-Jacobian trick — the product on the right is over just the diagonal entries, exactly as in section 1.13's observation about triangular matrices.
Let z = f(x) be an invertible, differentiable transformation, with z distributed according to a known simple density p_Z. For any small region dx around a point x, the transformed region has volume |det J_f(x)|·dx (section 1.12's geometric meaning of the determinant — it's exactly the local volume-scaling factor of the map). Conservation of total probability requires the probability mass in the two corresponding regions to match exactly:
Cancel the shared infinitesimal volume element dx from both sides and take logs:
This is exactly the formula stated above, and it makes clear why the determinant term isn't optional or a correction factor — it's the exact bookkeeping needed to keep p_X a valid density (integrating to 1) after a change of coordinates that locally stretches or shrinks volume by a different amount at every point.
Where this is used: this is the training objective itself for every normalizing flow model — maximizing log p_X(x) on real data requires evaluating exactly this formula for every training example, which is precisely why the triangular-Jacobian architectural trick (making the determinant cheap) is not an optimization but a hard requirement for the model to be trainable at all.
A simple "coupling layer" (the building block of RealNVP) splits the input into two halves, x = (x₁, x₂), and outputs y₁ = x₁ (unchanged) and y₂ = x₂ · exp(s(x₁)) + t(x₁), where s and t are arbitrary neural networks. The Jacobian of this map is exactly lower-triangular by construction — y₁ doesn't depend on x₂ at all — with diagonal entries 1 (from y₁) and exp(s(x₁)) (from y₂). The log-determinant is therefore simply sum(s(x₁)) — a trivial sum, computed directly from the network's own output, with no matrix operations of any kind required.
Notice there's no np.linalg.det call anywhere — that's the whole point of designing the transformation this way.
import numpy as np
def coupling_forward(x, s_fn, t_fn):
x1, x2 = x[:, 0], x[:, 1]
s = s_fn(x1) # any function/network of x1
y1 = x1
y2 = x2 * np.exp(s) + t_fn(x1)
log_det = s # log|det J| = sum of log-diagonal = s, here
return np.stack([y1, y2], axis=1), log_det
x = np.random.randn(5, 2)
y, log_det = coupling_forward(x, s_fn=lambda x1: 0.1 * x1, t_fn=lambda x1: 0.5 * x1)
print(y)
print(log_det) # the ENTIRE Jacobian determinant computation, no matrix needed- RealNVP and Glow use stacks of coupling layers exactly like the one above to build expressive, exactly-invertible generative models for images.
- Masked Autoregressive Flow (MAF) achieves the same triangular-Jacobian property by making each output dimension depend only on previous dimensions in a fixed order — a different architectural route to the identical mathematical guarantee.
- Variational inference uses normalizing flows to build flexible approximate posterior distributions that remain exactly tractable to evaluate and sample from.
- Designing a flow layer without checking that its Jacobian is actually triangular (or otherwise cheap to compute) — this is the single design constraint that makes normalizing flows tractable at all; violating it reintroduces the full
O(n³)determinant cost this whole architecture exists to avoid. - Forgetting the absolute value in
log|det J|— a negative Jacobian determinant is perfectly valid (it just means the transformation includes a reflection), but the log-likelihood formula specifically needs the magnitude.
Going deeper
Coupling layers alone would only ever transform half the input, since y₁ = x₁ exactly — real architectures alternate which half is held fixed across layers, so that after enough layers, every dimension has eventually been transformed by every other.
At the master level: continuous normalizing flows (built on Neural ODEs) replace this discrete stack of triangular-Jacobian layers with a continuous-time transformation whose log-determinant is instead computed via the trace (section 1.23) of the instantaneous Jacobian, integrated over time — the "instantaneous change of variables" formula — connecting this entire lesson back to trace, matrix calculus (section 1.17), and differential equations in one further generalization.
An entire modern generative modeling paradigm exists because someone asked "how do we design a neural network layer whose Jacobian is cheap to determinant?" — and the answer, triangularity, was sitting in this chapter the whole time.