Stable Diffusion
Denoising diffusion · Aug 2022
Generates images by learning to remove noise. Training adds noise to images step by step; the network learns to predict that noise, and generation runs the process backwards from pure static.
The statistical view
The noise that was added to a compressed image at step t, given the caption. Predicting the noise is equivalent to learning the score: the direction that makes a noisy sample more likely under the data distribution.
xₜ = √ᾱₜ · x₀ + √(1 − ᾱₜ) · ε, ε ~ N(0, I)
- Loss
- L = E‖ε − ε_θ(xₜ, t, text)‖²: plain mean-squared error between the true and the predicted noise, at a random step t.
- Answering
- Start from Gaussian noise and denoise over 20 to 50 steps. Classifier-free guidance pushes the result towards the prompt by extrapolating from the unconditioned prediction to the conditioned one.
What noise was added?
Destroy the data with Gaussian noise in small steps, learn to undo one step, then run the whole chain backwards to sample something new.
A demonstration of the objective, computed in your browser on illustrative data. It is not Stable Diffusion running here.
How it is trained
- 1Compress each image into a latent grid 8 times smaller per side with an autoencoder.
- 2Pick a random step t and add noise to the latent: the schedule ᾱₜ decides how much signal survives.
- 3Train the U-Net to predict the added noise, given t and the caption’s text embedding.
- 4Drop the caption part of the time, so one network learns both conditioned and unconditioned predictions for guidance.
- 5Generate: denoise from pure noise, then decode the latent back to pixels.
Line of descent
- Jun 2020DDPM — Ho et al. make diffusion competitive with GANs on image quality.
- Dec 2021Latent diffusion — Rombach et al. (LMU Munich, Runway): diffuse in a compressed space.
- Aug 2022Stable Diffusion v1 — Weights released publicly.
- Mar 2024Stable Diffusion 3 — Rectified flow and a Transformer replace the U-Net.
What data teams use it for
- Synthetic images to augment small or imbalanced training sets
- Research on privacy-preserving synthetic data
- Inpainting: filling missing or masked regions
- Illustrations for reports
Limits
- Web-scraped training data raises copyright and consent questions
- Synthetic data inherits, and can amplify, the biases of the training data
- Early versions struggle with hands, lettering and exact counts
- Many denoising steps make generation slower than one forward pass
Classic algorithms underneath
The same ideas, small enough to train in a browser.
