AI-101 Day 42: Diffusion, why image models work nothing like text models
Everything since Day 32 has described the same machine: one token at a time, left to right, each one conditioned on all the text before it.
Image generation is not that.
🌫 It starts from noise and cleans it up. Training is easiest understood backwards: take a real image, add a little Gaussian noise, then more, then more, until it is indistinguishable from static. Then train a network to do one very simple thing, given a noisy image and which step it is on, predict *the noise that was added*. The loss is the squared error between predicted and actual noise. That is the whole objective.
Generating is that run in reverse: start from pure random numbers and subtract the predicted noise, N times. DDPM (2020) needed 1,000 steps; modern samplers do 20–50. Crucially, every step moves the entire image at once, nothing is drawn left to right.
📝 Why text can't do this. Text is discrete. A pixel can be 137 or 138 and 137.3 still means something; there is nothing between "cat" and "dog". You cannot add a little bit of noise to a word. (Discrete diffusion for text is a live research area, but today's mainstream language models still emit one token at a time.)
🗜 The trick that made it practical. A 512×512 colour image is 786,432 numbers to process on every one of those steps. Latent Diffusion, the Stable Diffusion paper, 2022, compresses the image first, to 64×64×4 = 16,384 numbers, runs the diffusion in that latent space, and decodes back to pixels at the end. 48× smaller. That is the step that moved image generation from a data centre to a consumer graphics card.
Crucially, every step moves the entire image at once, nothing is drawn left to right.
🎯 How your prompt gets in. The text becomes vectors via a text encoder and enters each denoising step through cross-attention, Day 32's attention again, this time with the image attending to the words. Then classifier-free guidance: run the model twice per step, once with your prompt and once with nothing, and push harder in the direction of the prompted one. How hard you push is the guidance / CFG scale you see in the UI. Too low and it ignores you; too high and you get those over-saturated, deep-fried images, not a bug, just pushed too far.
🧩 Three worth keeping:
• 🖼 Images can't stream. Text arrives word by word; an image makes you wait and then appears whole. That isn't an interface decision, until the final step, the picture is still noise
• ✏️ Editing is the same operation. Start from your photo with *half* the noise added rather than from pure static, then denoise: that's img2img, and the "strength" slider is simply how much noise you added
• 🔢 Steps buy time, not quality. Past a point, more steps change almost nothing. What decides the result is the model and your prompt
📌 Takeaway: a language model assembles an answer one piece at a time; an image model lets the whole picture surface out of noise. Different machines, which is why they fail in different ways.
