By Gopi Krishna Tummala
How an image or video diffusion model goes from raw web data to a product: curating and re-captioning data, pre-training from low to high resolution, then quality tuning, preference tuning and reinforcement learning.
Part 5 of 8 in the diffusion series. Previous: Video Diffusion. Next: Diffusion for Action.
The first four parts explained what a diffusion model is and how it generates. None of that produces a good fox picture without the right training. A model trained only on raw web images with their raw captions can draw a fox, but it may not put it in snow when asked, may make it look like a stock photo from 2009, and may fail entirely at “at night.”
Training happens in three phases that improve different things. Data preparation decides what the model can know. Pre-training teaches it what the world looks like. Post-training teaches it what people want. This post walks through each, with the techniques that labs have published.
The idea in one picture
flowchart TD
subgraph Data["Data: what the model can know"]
W["Billions of web images<br/>and video clips"] --> F["Filter<br/>safety, quality, duplicates"]
F --> C["Re-caption<br/>a VLM writes detailed captions"]
end
subgraph Pre["Pre-training: what the world looks like"]
LR["Low resolution, huge volume"] --> HR["Higher resolution,<br/>more selective data"]
end
subgraph Post["Post-training: what people want"]
Q["Quality tuning<br/>thousands of excellent images"] --> P["Preference tuning<br/>better vs worse pairs"]
P --> R["Reinforcement learning<br/>against reward models and checkers"]
end
C --> LR
HR --> Q
R -.->|"failures and user feedback<br/>become new data"| WEach phase uses less data than the one before and changes the model less, but post-training is often what users notice most.
Data: most of the work
Image models were first trained on web-scale image-text pairs. LAION-5B, with 5.85 billion pairs scraped from the web, made open models like Stable Diffusion possible. It also showed the risks of scale: in December 2023 it was taken offline after researchers found links to child sexual abuse material, and it was re-released in 2024 as a cleaned version, Re-LAION-5B. Filtering isn’t a nicety.
Two practices now define a good data pipeline.
Filtering in stages. Remove unsafe content, duplicates, tiny or corrupt images, and low-quality ones using learned aesthetic and quality scores. Google’s Imagen 3 paper also removes AI-generated images, so the model doesn’t learn their artifacts. For video, add scene-cut splitting, motion scoring and removal of static clips; Stable Video Diffusion showed that a carefully curated video set beat simply using more video.
Re-captioning. Web captions are short, wrong or empty: a fox photo might be labeled “IMG_4521.jpg.” OpenAI’s DALL·E 3 report showed that replacing them with detailed synthetic captions from a captioning model sharply improved how well the model follows prompts. Now nearly every lab uses a vision-language model to write a detailed caption, often several of different lengths, for every training image or clip. That’s also why production systems rewrite short user prompts before generation: to match the long captions the model was trained on.
The Multimodal Atlas covers these data pipelines in more detail.
Pre-training: low resolution first
Pre-training is where the model learns what foxes, snow and night look like, and how text maps to them. Two choices shape it.
A resolution curriculum. Training starts at low resolution, often 256×256, which has a sixteenth of the pixels of 1024×1024 and so is far cheaper per image. Composition and concepts are learned there, on huge volumes of data. Later stages raise the resolution on smaller, more selective data, where the model learns fur and snow texture. Video models add a step before that: train on images first, then mix in short low-resolution clips, then longer and sharper ones, as Meta’s Movie Gen describes.
Weighting noise levels. Every training example uses a randomly chosen noise level, and how those levels are weighted matters. Very noisy steps are nearly impossible to learn from, and nearly clean ones are trivially easy; the middle carries most of the useful signal. Min-SNR weighting (Hang et al., 2023) caps how much the easy, low-noise steps can dominate the loss, which they found made training converge several times faster. Stable Diffusion 3’s rectified flow work similarly samples middle noise levels more often. A related fix from Part 1 ensures the last step is pure noise, so the model can make a fox truly at night.
Large runs also use the infrastructure covered in the MLOps series: sharded training (Training Frameworks), streaming data pipelines (Ray Data is common for video curation), and frequent checkpoints.
Post-training: from capable to good
A pre-trained model knows a lot but has no particular taste. Post-training typically has three steps.
Quality tuning. Fine-tune on a small set of exceptional images. Meta’s Emu found that a few thousand carefully selected, highly aesthetic images noticeably improved visual appeal while keeping the model’s range. It’s the diffusion equivalent of supervised fine-tuning: a little very good data goes a long way.
Preference tuning. People find it easier to say which of two images is better than to describe a perfect one. Diffusion-DPO (Wallace et al., 2023) adapted Direct Preference Optimization, from language models, to diffusion. It trains on pairs of images for the same prompt where people preferred one, without a separate reward model. It used the Pick-a-Pic dataset of human preferences, and fine-tuning SDXL with it improved both visual appeal and prompt-following in human evaluations.
Reinforcement learning against rewards. When a quality can be scored automatically, the model can optimize it directly. DDPO (Black et al., 2023) treated each denoising step as an action and optimized rewards like aesthetic scores and prompt alignment. Flow-GRPO applied the group-relative approach from language models, explained in Post-Training, to flow-matching models, using checkers like object detectors that can verify “one fox, in snow.” Reward models trained on human ratings, such as ImageReward, are common signals.
The danger in this last step is reward hacking. Optimize a single aesthetic score and every fox comes out oversaturated and glossy. Teams use several rewards at once, keep the model close to where it started, and check samples by eye.
Watching training health
A few signals catch most problems early:
- Loss and gradient norm. A sudden spike usually means instability. Gradient clipping, a learning-rate warmup and lower learning rates are the standard responses.
- EMA weights. Most diffusion models generate from an exponential moving average of the weights rather than the latest weights, which gives smoother, better samples. Evaluate the EMA model, not the raw one.
- Fixed-prompt samples. Generate the same prompts and seeds at regular checkpoints and look at them. Automated metrics like FID move slowly; a sudden shift in color or composition is visible immediately.
What I’d remember
- Data decides what’s possible. Filter hard, and re-caption everything with a VLM.
- Pre-train cheap, then sharp. Low resolution and huge volume first, then higher resolution on better data; images before video.
- Weight the useful noise levels. Min-SNR and middle-heavy sampling speed up training.
- Post-train in three steps. Quality tuning, preference tuning, then reinforcement learning against rewards.
- Beware reward hacking. Several rewards, a leash to the original model, and human spot checks.
Where to go next
- Diffusion for Action, the next part: the same recipe applied to robot motion.
- The Multimodal Atlas, for this lifecycle next to those of vision-language and video models.
- Papers: Emu, Diffusion-DPO, DDPO, Min-SNR.
Previous: Video Diffusion. Next: Diffusion for Action.
Questions to sit with
- Why does re-captioning training data change what a user’s prompt needs to look like?
- Why might a few thousand curated images change a model more visibly than a billion more web images?
- If every image starts looking glossy and oversaturated after RL, what went wrong, and what would you change?