A diffusion model is a generative AI model that creates data by starting from random noise and removing that noise step by step until a clear output appears. It learns this by studying how real images become noisy and then training a network to reverse the process. Diffusion models power most modern text-to-image tools and many video generation systems.
Key takeaways
- Training adds noise to real data; the model learns to predict and remove it.
- Generation runs in reverse: noise in, image out, over many small steps.
- A text prompt guides each step, which is why diffusion suits prompt-based generation.
- Latent diffusion works in a compressed space, making it far cheaper.
- Diffusion has largely replaced GANs for image generation because it is more stable and diverse.
A diffusion model explained in simple terms
Imagine a sculptor who can see a statue inside a block of marble and chips away the excess. A diffusion model sees an image “inside” a field of random static and removes the static a little at a time. Early steps settle the rough layout and colours. Later steps add edges, textures and fine detail. The prompt tells it which statue to look for.
How a diffusion model works
Training (forward process):
- Take a real image.
- Add a small amount of random noise, then more, over many steps, until it is pure noise.
- At random steps, show the network the noisy image and ask it to predict the noise that was added.
- Adjust the network to predict noise more accurately, using a loss function and backpropagation.
Generation (reverse process):
- Start with pure random noise.
- The network predicts the noise present and removes part of it.
- Repeat for dozens of steps (older methods used hundreds or more).
- The result is a new image that looks like it came from the training data.
Guiding the output with text
In text-to-image systems, a text encoder turns the prompt into vectors. At each denoising step, the network is conditioned on those vectors. A technique called classifier-free guidance compares the model’s prediction with and without the prompt and pushes the result towards the prompt. The guidance scale setting controls how strictly the image follows the prompt: higher values follow more closely but can reduce naturalness.
Latent diffusion
Running diffusion on full-size pixels is expensive. Latent diffusion, described by Robin Rombach and colleagues and released as Stable Diffusion in 2022, first uses an autoencoder to compress images into a much smaller latent representation. Diffusion runs in that latent space, and a decoder turns the final latent back into pixels. This made high-quality generation possible on consumer GPUs.
A brief history
| Year | Milestone |
|---|---|
| 2015 | Jascha Sohl-Dickstein and colleagues propose diffusion-based generative modelling |
| 2020 | “Denoising Diffusion Probabilistic Models” by Jonathan Ho and colleagues shows high-quality results |
| 2021 | Research shows diffusion can beat GANs on image quality benchmarks |
| 2022 | DALL·E 2, Imagen, Midjourney and Stable Diffusion bring text-to-image to the public |
| 2024 | Diffusion transformers drive a wave of high-quality text-to-video models |
Diffusion vs GANs
| Aspect | Diffusion | GAN |
|---|---|---|
| Training | Stable, simple objective | Unstable adversarial game |
| Diversity | High | Risk of mode collapse |
| Speed | Multiple steps (faster samplers help) | Single pass, very fast |
| Prompt control | Strong | Limited |
What diffusion can do beyond text-to-image
- Image editing and inpainting: regenerate part of an image while keeping the rest.
- Image-to-image: transform a sketch or photo guided by a prompt.
- Upscaling: add detail to low-resolution images.
- Video: generate consistent frames over time.
- Audio and music: generate sound from text descriptions.
- Science: design molecules and protein structures.
Limitations
- Slower generation than single-pass models, though distillation methods cut steps sharply.
- Struggles historically with text inside images, hands and exact counts, though newer models are much better.
- Can reproduce styles, biases and occasionally near-copies of training images, raising copyright concerns.
- High compute cost to train.