Diffusion Models: Learning to Reverse Destruction

2022: the year you could type a picture

In the spring of 2022 one image was suddenly everywhere: an astronaut riding a horse, photorealistic, summoned out of a typed sentence in about ten seconds. Within a few months DALL·E 2, Midjourney, and Stable Diffusion had all shipped, and the last one released its weights openly and ran on a gaming GPU.

For most of the previous decade the dominant way to make a network paint was the GAN, a generator and a discriminator locked in an adversarial game. GANs could be spectacular, and they were famously temperamental: unstable to train, prone to mode collapse, quietly learning to paint a handful of things in the dataset while ignoring the rest. What dethroned them was an idea that sounds backwards when you say it out loud. To teach a model to create images, first teach it to destroy them.

This post derives that idea from the probability up. By the end, this line, which is the entire training objective of a state-of-the-art image generator, should read as a consequence of the setup rather than a lucky guess:

$$L_{\text{simple}}(\theta) \;=\; \mathbb{E}_{t,\,x_0,\,\varepsilon}\Big[\,\big\lVert\, \varepsilon - \varepsilon_\theta(x_t, t)\,\big\rVert^2\,\Big].$$

We keep one thing in hand the whole way: the picture of an astronaut riding a horse, and what it takes to pull it out of static.

What “generate an image” actually asks for

Strip away the interface and generation is a statement about probability. A $512\times512$ colour image is a point in $\mathbb{R}^{d}$ with $d = 512\times512\times3 = 786{,}432$. Almost every point in that space is coloured snow. The images a human would call real occupy a vanishingly thin, intricately curved manifold of essentially zero volume, and we write the distribution over it as $p_{\text{data}}(x)$.

To generate is to draw a fresh sample $x \sim p_{\text{data}}$: a point absent from the training set that could plausibly have come from the same source. The obstacle is that $p_{\text{data}}$ has no closed form. We can’t write its density, we can’t normalise it, and we can’t sample from it directly. Every generative model is a strategy for getting around that one wall.

A GAN trains $G_\theta: \mathbb{R}^k \to \mathbb{R}^d$ to leap from a random seed straight onto the manifold, refereed by a discriminator. A VAE learns a smooth latent space but optimises a bound that tends to average, producing blurry samples. Both are trying to learn a single map from a simple Gaussian onto a thin, wildly multimodal manifold. Such a map has to stretch and fold space violently, and near the boundaries between modes it approaches discontinuity. Forcing that through an adversarial game is what made GANs so fragile.

So the question that unlocks everything: what if we didn’t have to do it in one leap?

The asymmetry worth exploiting

Corrupting an image is free. Add Gaussian noise, repeat, and after enough steps nothing is left but static. That operation has no parameters and needs no learning.

Reversing it over a large step is intractable. But over an infinitesimal step it has a known functional form, and that is the hinge the whole field turns on. For a Gaussian forward step with small enough variance, the reverse conditional is also Gaussian (a classical result about diffusion processes, and the observation Sohl-Dickstein et al. built on in 2015). So the plan becomes:

1. Define a fixed forward chain $q$ that destroys an image over $T$ steps.
2. Learn only the mean of each small reverse step, since its family is known to be Gaussian.
3. Chain those learned reverse steps from pure noise back to a sample.

The model only ever learns to denoise slightly, and we apply it a thousand times. Everything else in this post is machinery in service of that.

Forward: destroying an image in closed form

Call the clean image $x_0 \sim p_{\text{data}}$. The forward process is a fixed Markov chain producing $x_1, \dots, x_T$ (typically $T = 1000$), each step scaling down what is there and adding a dab of Gaussian noise:

$$q(x_t \mid x_{t-1}) \;=\; \mathcal{N}\!\big(x_t;\ \sqrt{1-\beta_t}\,x_{t-1},\ \beta_t I\big), \qquad q(x_{1:T}\mid x_0) = \prod_{t=1}^{T} q(x_t\mid x_{t-1}).$$

The variances $\beta_1, \dots, \beta_T \in (0,1)$ are the noise schedule. The scaling factor $\sqrt{1-\beta_t}$ earns its place by keeping the process variance-preserving. Track the per-coordinate variance $v_t$ and, because independent Gaussians add in variance, the two coefficients enter as their squares:

$$v_t \;=\; (1-\beta_t)\,v_{t-1} + \beta_t \qquad\Longrightarrow\qquad v_t - 1 \;=\; \alpha_t\,(v_{t-1}-1).$$

Those weights sum to one, so the update is a weighted average and $v=1$ is a fixed point. Better, it is an attracting one: any deviation shrinks by $\alpha_t$ per step and compounds to $v_T - 1 = \bar\alpha_T(v_0-1)$. The mean is meanwhile multiplied by $\sqrt{\alpha_t}$ each step, so it decays to zero on the same schedule, and $q(x_T)$ lands on $\mathcal{N}(0,I)$ whatever image we started from. Drop the scaling and you would get $v_T = v_0 + \sum_t \beta_t$ instead, which grows without bound. That variance-exploding convention is a legitimate alternative and is what Song & Ermon’s NCSN uses, but then the network has to cope with inputs spanning orders of magnitude, and you need to know $\sum_t\beta_t$ to know where sampling starts.

Sampling from that Gaussian directly would leave us with no gradient path back to $x_0$, so we use the reparameterization trick (Kingma & Welling, 2014): write the sample as a deterministic function of the parameters plus an independent noise draw. With $\alpha_t := 1 - \beta_t$,

$$x_t \;=\; \sqrt{\alpha_t}\,x_{t-1} \;+\; \sqrt{1-\alpha_t}\;\varepsilon_{t-1}, \qquad \varepsilon_{t-1} \sim \mathcal{N}(0, I).$$

Now the first useful consequence. You might think reaching $x_t$ requires grinding through all $t$ steps. It doesn’t, because a scale-and-add-Gaussian composes with itself. Expand two steps:

$$x_t \;=\; \sqrt{\alpha_t \alpha_{t-1}}\;x_{t-2} \;+\; \underbrace{\sqrt{\alpha_t(1-\alpha_{t-1})}\;\varepsilon_{t-2} \;+\; \sqrt{1-\alpha_t}\;\varepsilon_{t-1}}_{\text{two independent zero-mean Gaussians}}.$$

Independent Gaussians add in variance, so those two terms merge into a single draw with variance $\alpha_t(1-\alpha_{t-1}) + (1-\alpha_t) = 1 - \alpha_t\alpha_{t-1}$. Induction on that step gives the closed form. Writing $\bar\alpha_t := \prod_{i=1}^{t}\alpha_i$,

$$q(x_t \mid x_0) \;=\; \mathcal{N}\!\big(x_t;\ \sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t) I\big) \quad\Longleftrightarrow\quad \boxed{\,x_t \;=\; \sqrt{\bar\alpha_t}\,x_0 \;+\; \sqrt{1-\bar\alpha_t}\,\varepsilon\,}, \quad \varepsilon \sim \mathcal{N}(0,I).$$

That boxed line does more work than any other equation in the post. It says we can jump to any noise level in one operation, which makes training over random timesteps cheap. It is a dial between two ingredients: $\sqrt{\bar\alpha_t}$ is how much signal survives and $\sqrt{1-\bar\alpha_t}$ is how much noise has been mixed in. Their ratio is the signal-to-noise ratio of the corrupted image,

$$\text{SNR}(t) \;=\; \frac{\bar\alpha_t}{1-\bar\alpha_t},$$

which falls monotonically from very large to nearly zero. Put our astronaut in as $x_0$ and the SNR tells you what is left of the rider: at $t=250$ the silhouette is grainy but unmistakable, and by $t=800$ nothing survives but a dark smudge against the glow. As $t \to T$ we want $\bar\alpha_T \approx 0$, so $q(x_T\mid x_0) \approx \mathcal{N}(0,I)$ regardless of $x_0$. That is the guarantee that lets us start generation from pure Gaussian noise: the forward process provably ends there, whatever we fed it.

Drag the slider and watch the astronaut dissolve, with both coefficients tracking your position:

Choosing the schedule

Since $\{\beta_t\}$ is ours to pick, the question is how to spend a thousand steps well. DDPM used a linear schedule, $\beta_t$ rising from $10^{-4}$ to $0.02$. Its $\bar\alpha_t$ crashes to near zero well before $t = T$, so hundreds of late steps carry no signal and teach the network almost nothing. Nichol & Dhariwal proposed defining the schedule through $\bar\alpha_t$ directly:

$$\bar\alpha_t \;=\; \frac{f(t)}{f(0)}, \qquad f(t) \;=\; \cos^2\!\left(\frac{t/T + s}{1+s}\cdot\frac{\pi}{2}\right), \qquad s = 0.008,$$

with $\beta_t = 1 - \bar\alpha_t/\bar\alpha_{t-1}$ clipped for stability. The cosine schedule declines gently enough to keep signal alive across the whole chain. Count the steps that sit below an SNR of $0.01$, which is static a network can learn almost nothing from: under the linear schedule that is about a third of all $T$ steps, and under cosine it is six percent. At $t = 750$ the linear schedule is down to an SNR of $0.003$ while cosine still holds $0.17$. Same idea, better budgeting.

There is a nicer way to read that formula. Since $\bar\alpha_t + (1-\bar\alpha_t) = 1$ identically, the two coefficients in the boxed equation are always a point on the unit circle, and under this schedule they are exactly the cosine and sine of $\theta_t = \frac{t/T+s}{1+s}\cdot\frac{\pi}{2}$:

$$x_t \;=\; \cos(\theta_t)\,x_0 \;+\; \sin(\theta_t)\,\varepsilon.$$

The forward process rotates $x_0$ toward $\varepsilon$ through a quarter circle, and the cosine schedule is the one that sweeps it at constant angular speed. That is not a figure of speech: it advances $8.9°$ per hundred steps from start to finish, while the linear schedule opens at $18.7°$ and closes at $0.6°$, covering 57% of the arc in its first 300 steps against cosine’s 31%. Every schedule rides the same circle, so choosing one is only choosing a pace. Hold onto that picture: rectified flow, at the end of this post, is the idea of taking the chord instead of the arc. Flip between the schedules, and switch on the rotation view:

Backward: the posterior is tractable if you cheat

To generate we need the reverse conditional $q(x_{t-1}\mid x_t)$, and here we hit the wall from earlier. Computing it exactly requires marginalising over $p_{\text{data}}$, the thing we cannot write down. So we approximate it with a network, using the guarantee that for small $\beta_t$ the true reverse step is Gaussian:

$$p_\theta(x_{t-1}\mid x_t) \;=\; \mathcal{N}\!\big(x_{t-1};\ \mu_\theta(x_t, t),\ \Sigma_\theta(x_t, t)\big), \qquad p_\theta(x_{0:T}) = p(x_T)\prod_{t=1}^{T} p_\theta(x_{t-1}\mid x_t).$$

What should $\mu_\theta$ regress toward? Here is the algebraic turn that makes the whole thing trainable. The reverse step is intractable in general, but it becomes an exact Gaussian once you condition on the clean image $x_0$. Applying Bayes’ rule to the forward chain,

$$q(x_{t-1}\mid x_t, x_0) \;=\; q(x_t \mid x_{t-1}, x_0)\,\frac{q(x_{t-1}\mid x_0)}{q(x_t \mid x_0)} \;=\; q(x_t \mid x_{t-1})\,\frac{q(x_{t-1}\mid x_0)}{q(x_t \mid x_0)},$$

where the second equality is the Markov property of the forward chain. Every factor on the right is now a Gaussian we already have in closed form: one from the definition of the chain, two from the boxed marginal. Multiplying them out and collecting the terms quadratic in $x_{t-1}$ gives a Gaussian $\mathcal{N}(x_{t-1}; \tilde\mu_t(x_t, x_0), \tilde\beta_t I)$ with

$$\tilde\beta_t \;=\; \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\,\beta_t, \qquad \tilde\mu_t(x_t, x_0) \;=\; \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\,x_0 \;+\; \frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\,x_t.$$

Read that mean as a fixed, known interpolation between where we are ($x_t$) and where we started ($x_0$). Both coefficients are constants from the schedule. The only unknown at sampling time is $x_0$, which is exactly the astronaut we are trying to conjure and obviously don’t have.

Shrink $p_{\text{data}}$ down to four points on a line and all of this can be drawn instead of asserted. Both densities are exact here, so the comparison is the real one:

That picture is the shape of the whole difficulty. The intractable object is a sum over every $x_0$ that could have produced $x_t$, and every single term in that sum is a Gaussian we can write down. What we cannot do is enumerate the terms.

So the network’s job collapses to estimating the clean image. And now invert the boxed forward equation, $x_0 = \frac{1}{\sqrt{\bar\alpha_t}}\big(x_t - \sqrt{1-\bar\alpha_t}\,\varepsilon\big)$, and substitute it into $\tilde\mu_t$. The $x_t$ terms combine and the mean simplifies dramatically:

$$\tilde\mu_t \;=\; \frac{1}{\sqrt{\alpha_t}}\left(x_t \;-\; \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\varepsilon\right).$$

Everything on the right is known except $\varepsilon$. Estimating the clean image and estimating the added noise are the same task in different coordinates, and predicting $\varepsilon$ turns out to be far better conditioned to train, since the target has fixed unit variance at every timestep while $x_0$ does not. So the network becomes a noise predictor $\varepsilon_\theta(x_t, t)$, and its induced mean is

$$\mu_\theta(x_t, t) \;=\; \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\varepsilon_\theta(x_t,t)\right).$$

The objective: a page of KL that collapses to one squared error

The likelihood $p_\theta(x_0) = \int p_\theta(x_{0:T})\,dx_{1:T}$ is intractable, so we optimise the variational bound, exactly as in a VAE with the forward chain playing the role of a fixed encoder:

$$-\log p_\theta(x_0) \;\le\; \mathbb{E}_{q}\!\left[\log \frac{q(x_{1:T}\mid x_0)}{p_\theta(x_{0:T})}\right] \;=:\; L_{\text{VLB}}.$$

Rewriting the forward chain in its $x_0$-conditioned form and telescoping the products, this decomposes into per-timestep terms:

$$L_{\text{VLB}} \;=\; \underbrace{D_{\mathrm{KL}}\big(q(x_T\mid x_0)\,\|\,p(x_T)\big)}_{L_T} \;+\; \sum_{t=2}^{T} \underbrace{D_{\mathrm{KL}}\big(q(x_{t-1}\mid x_t, x_0)\,\|\,p_\theta(x_{t-1}\mid x_t)\big)}_{L_{t-1}} \;\underbrace{-\;\log p_\theta(x_0\mid x_1)}_{L_0}.$$

Three observations flatten this. $L_T$ has no learnable parameters, since $q$ is fixed and $p(x_T) = \mathcal{N}(0,I)$, so it is a constant we can drop. $L_0$ is a discretised Gaussian likelihood over the final decoding step, a detail rather than the main event. And every $L_{t-1}$ is a KL between two Gaussians, which for a fixed variance $\sigma_t^2$ has a closed form that is just a scaled squared distance between means:

$$L_{t-1} \;=\; \mathbb{E}_q\!\left[\frac{1}{2\sigma_t^2}\big\lVert \tilde\mu_t(x_t, x_0) - \mu_\theta(x_t, t) \big\rVert^2\right] + C.$$

Substitute the two mean expressions from the last section. The $x_t/\sqrt{\alpha_t}$ terms cancel outright, leaving only the noise difference scaled by the schedule constants:

$$L_{t-1} \;=\; \mathbb{E}_{x_0, \varepsilon}\!\left[\frac{\beta_t^2}{2\sigma_t^2\,\alpha_t\,(1-\bar\alpha_t)}\,\big\lVert \varepsilon - \varepsilon_\theta(x_t, t)\big\rVert^2\right].$$

The variational bound is now a weighted denoising regression. Ho et al.’s empirical finding was that dropping the weight improves sample quality: the coefficient is large at small $t$, over-emphasising imperceptible high-frequency detail, and setting it to 1 reweights training toward the harder, more semantically important high-noise steps. What survives is:

$$L_{\text{simple}}(\theta) \;=\; \mathbb{E}_{t\sim\mathcal{U}\{1,T\},\,x_0,\,\varepsilon}\Big[\big\lVert \varepsilon - \varepsilon_\theta\big(\underbrace{\textstyle\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon}_{x_t},\, t\big)\big\rVert^2\Big].$$

Which is the whole training loop:

1. Sample $x_0 \sim p_{\text{data}}$, $t \sim \mathcal{U}\{1,\dots,T\}$, $\varepsilon \sim \mathcal{N}(0,I)$.
2. Form $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon$ in one operation.
3. Take a gradient step on $\lVert \varepsilon - \varepsilon_\theta(x_t,t)\rVert^2$.

Sampling runs the learned mean with fresh noise injected, which is the DDPM ancestral sampler:

$$x_{t-1} \;=\; \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{1-\alpha_t}{\sqrt{1-\bar\alpha_t}}\,\varepsilon_\theta(x_t,t)\right) \;+\; \sigma_t z, \qquad z\sim\mathcal{N}(0,I),$$

with $z = 0$ at $t=1$ and $\sigma_t^2$ set to either $\beta_t$ or $\tilde\beta_t$ (both work; the two are the upper and lower bounds on the true reverse variance).

No adversary, no minimax balance, no mode collapse. Sampling from an intractable distribution has been reduced to self-supervised regression: add known noise, predict it back. Nothing in that loop knows what an astronaut is. The network sees corrupted images at every noise level and learns the statistics of what real data looks like underneath, which turns out to be enough.

That loop is short enough to be suspicious, so here it is running on real pixels. The denoiser below is exact rather than learned: for a finite training set $\{x^1,\dots,x^K\}$ the optimal noise predictor has a closed form, because $q(x_t)$ is then a known mixture of $K$ Gaussians. Tweedie’s formula turns it into a softmax vote over the training images, each weighted by how well it explains what the sampler is currently holding:

$$\hat x_0 \;=\; \mathbb{E}[x_0 \mid x_t] \;=\; \sum_{i=1}^{K} w_i\,x^i, \qquad w_i \;=\; \frac{e^{-d_i}}{\sum_{j} e^{-d_j}}, \qquad d_i \;=\; \frac{\lVert x_t - \sqrt{\bar\alpha_t}\,x^i\rVert^2}{2\,(1-\bar\alpha_t)},$$

so $\varepsilon_\theta$ is exactly right and every pixel below is the genuine DDIM update rather than an animation of one. (The panel blurs each training point by a small width $\sigma_d$, which shifts $w$ and $\hat x_0$ a little and is what keeps the denoiser from being a pure lookup table.) Read the three images left to right as the boxed forward equation solved for $x_0$.

The right-hand panel is the one to watch. It holds the model’s running guess at the finished picture, and it sharpens out of a smear of everything the model knows into one specific image while no new information arrives. Then look at when that happens: the weights collapse onto a single training image around $t = 900$, where $\sqrt{\bar\alpha_t} \approx 0.15$ and $x_t$ is still indistinguishable from static. The subject gets settled in the first handful of high-noise steps, and the remaining nine hundred are rendering. That is the concrete reason the cosine schedule’s extra budget up at high $t$ is worth paying for, and the reason dropping the weight in $L_{\text{simple}}$ helps at all: both move effort to the steps where the decision is actually made.

What the network is really learning: a score field

Noise prediction works, and it looks like a trick until you see what $\varepsilon_\theta$ actually approximates. Take the log of the closed-form forward marginal and differentiate with respect to $x_t$:

$$\nabla_{x_t} \log q(x_t \mid x_0) \;=\; -\,\frac{x_t - \sqrt{\bar\alpha_t}\,x_0}{1-\bar\alpha_t} \;=\; -\,\frac{\varepsilon}{\sqrt{1-\bar\alpha_t}}.$$

Vincent’s 2011 denoising score matching result says that regressing on this conditional score is equivalent, up to a constant, to regressing on the marginal score $\nabla_{x_t}\log q(x_t)$. So the trained noise predictor gives us

$$s_\theta(x_t, t) \;:=\; -\,\frac{\varepsilon_\theta(x_t, t)}{\sqrt{1-\bar\alpha_t}} \;\approx\; \underbrace{\nabla_{x_t} \log q(x_t)}_{\text{the score}}.$$

The equivalent statement in the other direction is Tweedie’s formula, which says the posterior mean of a Gaussian-corrupted variable is recoverable from the score of the corrupted density:

$$\mathbb{E}[x_0 \mid x_t] \;=\; \frac{1}{\sqrt{\bar\alpha_t}}\Big(x_t + (1-\bar\alpha_t)\,\nabla_{x_t}\log q(x_t)\Big).$$

Denoising, predicting noise, and estimating the score are three names for one quantity.

That reframes generation completely. The score is a vector field: at every point in image space it points in the direction of steepest increase in data density. Generation becomes a walk up that field, with noise injected at each step so samples spread across the modes instead of collapsing onto the single most likely image. That walk is Langevin dynamics:

$$x_{i+1} \;=\; x_i \;+\; \frac{\delta}{2}\,\nabla_x \log p(x_i) \;+\; \sqrt{\delta}\,z_i, \qquad z_i \sim \mathcal{N}(0,I),$$

with step size $\delta$. Note what the update needs: only the score. The density $p$ itself, and its intractable normalising constant, never appear.

One thing that equation hides: there is no single field. We trained $\varepsilon_\theta$ on every noise level, so what the network holds is a whole family of score fields indexed by $t$, one per marginal $q(x_t)$. And they look nothing alike. Since $q(x_t)$ is the data mixture with every mean pulled toward the origin by $\sqrt{\bar\alpha_t}$ and every component widened to $\bar\alpha_t\sigma^2 + (1-\bar\alpha_t)$, at $t = T$ all the modes have blurred into one basin, and only as $t$ falls do they separate. That is the reason $\varepsilon_\theta$ takes a timestep as an argument at all, and the reason sampling anneals rather than descending a fixed landscape: it starts in the one basin that is trivial to find, and lets the basin split underneath it.

You can’t picture this in 786,432 dimensions, but in two you can see it. Below, a toy dataset lives in five clusters and the amber arrows are the score field. Drag $t$ first and watch the field morph: at $t=1000$ it is a single broad basin at the origin, the five clusters only resolve into separate basins around $t=500$, and by $t=400$ all five are distinct. Then hit play to sweep $t$ from 1000 down to 0 while a cloud of noise follows the field down. Because the arrows are the true annealed score, the cloud lands correctly distributed across the modes rather than piling onto their centres.

This is the most important picture in the post. Song et al. later showed that as $T \to \infty$ the discrete chain becomes a stochastic differential equation, $dx = f(x,t)\,dt + g(t)\,dw$. DDPM’s particular chain is the variance-preserving case, $f(x,t) = -\tfrac{1}{2}\beta(t)\,x$ and $g(t) = \sqrt{\beta(t)}$, which is the continuous-time reading of the $\sqrt{1-\beta_t}$ scaling we started with. Anderson’s 1982 result gives its reverse-time counterpart:

$$dx \;=\; \big[f(x,t) - g(t)^2\,\nabla_x \log p_t(x)\big]dt \;+\; g(t)\,d\bar w.$$

The score is the only unknown in that equation, and $\varepsilon_\theta$ is our estimate of it. Everything else is fixed by the schedule we chose.

Sampling: from a thousand steps to fifty

The same SDE has a deterministic twin with identical marginals at every $t$, the probability flow ODE:

$$\frac{dx}{dt} \;=\; f(x,t) - \tfrac{1}{2}g(t)^2\,\nabla_x \log p_t(x).$$

That is a consequential fact. Sampling is the solution of a smooth ODE, and smooth trajectories don’t need a thousand baby steps. You can take large strides across them with a decent solver.

DDIM is the discrete form of exactly this. Song et al. observed that the training objective only depends on the marginals $q(x_t\mid x_0)$, which pins down nothing about the joint. So they built a non-Markovian family sharing those marginals, with update

$$x_{t-1} \;=\; \sqrt{\bar\alpha_{t-1}}\;\underbrace{\left(\frac{x_t - \sqrt{1-\bar\alpha_t}\,\varepsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}}\right)}_{\text{predicted } x_0} \;+\; \underbrace{\sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\;\varepsilon_\theta(x_t,t)}_{\text{direction pointing to } x_t} \;+\; \sigma_t z.$$

The parameter $\sigma_t = \eta\sqrt{\tilde\beta_t}$ interpolates between the two regimes. At $\eta = 1$ this reduces to DDPM ancestral sampling. At $\eta = 0$ the stochastic term vanishes and sampling becomes deterministic, so the same trained network can be run on a sparse subsequence of timesteps, 50 of the 1000 or even 20, and still land on the manifold. No retraining is involved; the weights are identical.

The difference is easiest to see in the shape of the trajectory. DDPM’s is a random walk: the $\sigma_t z$ term knocks the point off course at every step and the next step spends itself undoing that, so the total distance travelled grows without bound as you add steps. DDIM’s trajectory is one fixed curve whose length barely changes with $N$, because $N$ only sets how finely you chop up a path that was already there. Both samplers are correct in the many-step limit; only one of them is cheap to shortcut. Drop the step count and switch samplers:

This is why you generate an image in 20 to 50 steps rather than 1000. Higher-order solvers like DPM-Solver++ push it lower, and distilled models collapse it to one. The 20× speedup that made these models usable came from noticing that the thing everyone stepped through a thousand times was a smooth curve.

Steering the walk: how text gets in

Everything so far generates some plausible image, and we asked for a specific one. Left alone the score field leads to any real image at all, and the odds it wanders to an astronaut on a horse are nil. We need to bend it toward the prompt.

The mechanism is conditioning: pass a text encoding $c$ into the noise predictor, $\varepsilon_\theta(x_t, t, c)$, which amounts to learning the conditional score $\nabla_x \log p(x\mid c)$. Conditioning alone tends to adhere loosely. The fix that made text-to-image genuinely obey is classifier-free guidance. During training the prompt is randomly dropped (roughly 10% of the time), so one network learns both a conditional and an unconditional predictor. At sampling time you extrapolate along the difference:

$$\tilde\varepsilon_\theta(x_t, c) \;=\; \varepsilon_\theta(x_t, \varnothing) \;+\; w\,\big[\varepsilon_\theta(x_t, c) - \varepsilon_\theta(x_t, \varnothing)\big].$$

In score terms the bracket is $\nabla_x \log p(c\mid x)$ by Bayes’ rule, so guidance is sampling from a distribution proportional to $p(x)\,p(c\mid x)^{w}$: the likelihood of the prompt, sharpened by an exponent. At $w=1$ you recover the honest conditional. Crank $w$ higher and the sample slams onto the prompt’s modes, at a real cost. Prompt fidelity rises while diversity falls, since a sharpened distribution has less mass in its tails. Stable Diffusion’s default of $w \approx 7.5$ is a deliberate compromise. Turn the dial and watch the trade:

That $c$ enters the network through cross-attention, the same mechanism that powers language models. Each patch of the image emits a query, the text tokens offer keys and values, and the image attends to the words. (If you have read the attention posts on this blog, it is that machinery pointed sideways from image to text.) The prompt reaches into every denoising step and bends the field.

Making it affordable: diffusion in a latent space

We now have a complete generator, and it is ruinously expensive. Every denoising step is a full pass of a large network over a 786,432-dimensional tensor, dozens of times per image.

Latent Diffusion, the “SD” in Stable Diffusion, applies the same instinct we have used throughout: do the expensive work where it is cheapest. Most of those 786,432 numbers encode imperceptible high-frequency detail rather than semantic content. So train an autoencoder $(\mathcal{E}, \mathcal{D})$ first, compressing $x$ into a latent $z = \mathcal{E}(x)$ and decoding it back with little perceptual loss. For Stable Diffusion a $512\times512\times3$ image becomes a $64\times64\times4$ latent: 786,432 numbers down to 16,384, a 48× reduction. The astronaut, the horse, and the lighting all survive that squeeze; the individual dust grains on the visor do not.

Then run the entire diffusion process in that space. The objective is unchanged except for where it lives:

$$L_{\text{LDM}} \;=\; \mathbb{E}_{\mathcal{E}(x),\,c,\,\varepsilon,\,t}\Big[\big\lVert \varepsilon - \varepsilon_\theta\big(z_t,\, t,\, \tau_\theta(c)\big)\big\rVert^2\Big],$$

where $\tau_\theta$ is the text encoder. The denoiser (a 865M-parameter U-Net in the original Stable Diffusion, increasingly a transformer in newer models) walks the score field over 16,384 numbers, and we decode to pixels once, at the end.

The panel below is a scale model of that arrangement: a real encoder, a latent you can actually look at, and the sampler from earlier running inside it. Flip the toggle to put the identical update back in pixel space, and notice that what changes is the cost rather than the result:

That single move is the difference between a research demo and something millions of people run on consumer GPUs.

The frontier

Everything above is the foundation, circa 2020 to 2021. The years since have been a story of unification and acceleration:

• One object, several languages. The DDPM noise-prediction view, the score-based SDE view, and flow matching describe the same underlying process. Predicting noise, predicting $x_0$, and predicting a velocity are reparameterizations of learning one vector field.
• Straight paths beat curved ones. DDPM’s trajectory through space is curved, which is part of why it needs many steps. Rectified flow and flow matching, the basis of Stable Diffusion 3 and Flux, train the model to follow straight paths from noise to data, and straight paths tolerate far bigger steps.
• Transformers replace the U-Net. The DiT treats latent patches as tokens, letting image generation ride the same scaling curves and the same FlashAttention kernels as large language models.
• From fifty steps to one. Consistency models, LCM, and the Turbo and Lightning families distil a slow teacher into a 1 to 4 step student, putting real-time generation in reach.

Each deserves its own deep dive, and each is easier once you can see generation as integrating a learned vector field.

Lessons

1. If a problem is impossible in one direction, check the other one, then break it up. Generating an image is brutal and corrupting one is free, so diffusion learns the free direction and inverts it. And nobody can map noise to a coherent image in one shot, but a thousand easy denoising steps compose into that impossible jump. Decomposition is a modelling strategy, not only an engineering one.
2. Push the intimidating objective all the way through. A variational bound over $T$ timesteps collapsed into one squared error, and the collapse was where the insight lived.
3. Then be willing to leave the theory behind. The objective people actually train departs from that bound, because dropping the weighting term improved samples. Theory got us to the right family, experiment picked the member.
4. Do the expensive work in the cheapest space. Latent diffusion kept the same denoiser and moved it into a space 48× smaller.
5. Look for the smooth structure hiding under the discrete one. A thousand sampling steps became twenty because someone recognised the sampler as an ODE. As with Flash Attention, the biggest wins come from seeing the true shape of the computation.

Conclusion

A diffusion model comes down to one inversion of intuition: to create, learn to destroy, then reverse the destruction. The forward process is a fixed Gaussian chain with a closed-form marginal. The reverse process is intractable in general and exactly Gaussian when conditioned on $x_0$, which is what turns a variational bound into a regression on noise. That noise predictor is a scaled score function, so sampling is integration of a learned vector field, steered by guidance and made affordable by running in a compressed latent.

Type “an astronaut riding a horse,” and a cloud of static begins its walk up the gradient.

References

1. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. — The $\varepsilon$-parameterization and the $L_{\text{simple}}$ objective.
2. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015. — The original forward/reverse diffusion formulation.
3. Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes. ICLR 2014. — The reparameterization trick and the variational bound this objective inherits.
4. Vincent, P. (2011). A Connection Between Score Matching and Denoising Autoencoders. Neural Computation 23(7). — Why regressing on the conditional score matches the marginal score.
5. Song, Y., & Ermon, S. (2019). Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS 2019. — Score matching with Langevin dynamics.
6. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021. — The SDE unification and the probability flow ODE.
7. Nichol, A., & Dhariwal, P. (2021). Improved Denoising Diffusion Probabilistic Models. ICML 2021. — The cosine schedule and learned variances.
8. Dhariwal, P., & Nichol, A. (2021). Diffusion Models Beat GANs on Image Synthesis. NeurIPS 2021. — Where guided diffusion overtook GANs on ImageNet FID.
9. Song, J., Meng, C., & Ermon, S. (2021). Denoising Diffusion Implicit Models. ICLR 2021. — DDIM and the non-Markovian sampling family.
10. Ho, J., & Salimans, T. (2022). Classifier-Free Diffusion Guidance. arXiv:2207.12598. — The guidance-scale knob.
11. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022. — Latent diffusion.
12. Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2022. — The EDM systematization of schedules, preconditioning, and samplers.
13. Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. (2023). Flow Matching for Generative Modeling. ICLR 2023. — Straight-path generation underlying SD3 and Flux.
14. Peebles, W., & Xie, S. (2023). Scalable Diffusion Models with Transformers. ICCV 2023. — DiT.