Why diffusion models struggle to predict noise through bottlenecks
The Noise-Transmission Burden of Diffusion Network: Why Image Prediction Can Outperform Noise Prediction · arXiv:2610.34834
In plain words
Diffusion models learn to turn a noisy image into a clean one. Some predict the added noise and subtract it from the input; if their input representation has thrown away information, they may be unable to cancel all that noise. This paper develops a theory of that problem and tests ways to reduce it.
In ImageNet experiments with a compressed input, adding a carefully chosen copy of the noisy image helped noise-predicting models denoise much better. The result is evidence about these tested models and settings, not a guarantee for every diffusion model.
The research
When a diffusion network compresses its input, what information must it keep to turn a noisy image into a clean one?
The answer may differ by prediction target: predicting noise requires the network to cancel corruption still present in the input, even when a compressed representation has lost some of that information.
The paper shows how input bottlenecks burden noise prediction and tests output preconditioning as a way to reduce the resulting denoising error.
The authors ask why predicting noise can work worse than predicting the clean image when a finite diffusion network has a compressed input. A noise predictor must cancel corruption carried by the noisy input, but compression can hide some of that noise from the network.
Their theory predicts an error floor from this lost information and shows how output preconditioning can reduce it. In bottlenecked ImageNet256 models, measured high-noise denoising error for direct noise prediction nearly matched the predicted floor, while exact preconditioning and image prediction had much lower error. A separate generation comparison showed a similar ordering.
The theory describes fixed noise levels and, for its width result, assumes the noisy patch can be recovered exactly through an affine readout. The experiments use one training run per configuration, and the theory does not show how denoising errors affect the final generated-image distribution.
What the evidence shows
- The finite-network question
This paper develops a theory and ImageNet experiments about why finite diffusion networks can perform differently when asked to predict a clean image or the noise added to it. Both prediction targets can yield the same optimal clean-image estimate in principle.
But a noise predictor must subtract its scaled noise estimate from the noisy input, so its network needs information to cancel the corruption.
- What compression discards
The theory asks what happens when an input bottleneck discards directions of a noisy image. For a linear encoder that retains r of d directions, the discarded Gaussian noise cannot be recovered by any downstream decoder.
The resulting clean-image error contribution grows with the noise level, while the optimal image predictor’s risk under the same bottleneck stays bounded.
- A shortcut changes the burden
Output preconditioning adds a scaled copy of the noisy input to the network output. In the theory, exact preconditioning cancels the contribution from discarded noise, while the EDM choice reduces it as noise grows.
In a bottlenecked JiT-B/16 model on ImageNet256, high-noise clean-image error for direct noise prediction was close to the predicted floor; exact preconditioning and image prediction had much lower error.
- Denoising and generation
The generation test points in the same direction. FID-10K compares generated-image statistics with reference-image statistics, with lower scores indicating closer agreement.
For the tested bottlenecked model, exact preconditioning scored 9.15 and image prediction 9.13, compared with 412.11 for direct noise prediction; EDM scored about 52.28. These are single-run measurements, so they do not show run-to-run uncertainty.
- Why width can help
For models with full input, the paper also studies width. Its theory assumes the hidden representation can recover the noisy patch through an affine readout; extra coordinates can then represent directions of the remaining denoising correction.
Wider ImageNet256 noise predictors had lower measured error in one comparison, while a proxy estimate found the low-noise correction spread across more directions than the high-noise correction.
Deep
- Original paper authors
- Gyeonghun Kang, Pengxi Liu, Bingshuo Qian, Xiang Cheng
- Submitted to arXiv
- License
- CC BY 4.0