A pixel-space image model gains from representing more scales
Scale as a Modality: Multi-Scale Pixel-Space Diffusion Transformers · arXiv:2611.08802
In plain words
Many image-making systems first shrink pictures into a compressed form. MS-pDiT instead works with pixels and looks at the same noisy image in several ways, from large patches to individual pixels. Its separate streams can share information, and the finest stream produces the image.
In an ImageNet test, adding a middle-sized view improved the match between generated images and real-image statistics compared with a two-view version. The comparison lasted 80 training epochs, so it does not tell us how adding more views would work in every setting.
The research
Image generators must decide how to represent fine detail alongside overall structure.
These results suggest that adding an intermediate image scale may be a more effective capacity choice than simply deepening a coarse-scale stream, changing what the ImageNet comparisons support about how pixel-space models can be scaled.
MS-pDiT generates images from pixels using parallel streams at several scales, whose two-way information exchange makes intermediate scales a useful capacity choice in ImageNet tests.
In an 80-epoch ImageNet 256x256 comparison, researchers added a 4x4-patch stream to a model with 16x16 and 1x1 streams. They compared generated images with reference statistics using FID, a distribution score where lower is better; the score dropped from 4.39 to 2.31.
A similar-parameter test also favored the intermediate scale over extra coarse-stream depth. Together, the comparisons support the authors’ view that representing more image granularities can use capacity effectively, while the added stream comparison alone does not isolate scale from model size.
The scale comparisons cover two to five scales in 80-epoch ImageNet 256x256 tests. They do not show whether adding scales would keep improving results beyond those configurations or training conditions.
What the evidence shows
- A model with parallel scales
MS-pDiT is a pixel-space diffusion transformer: it generates images directly from pixels and represents each noisy image in parallel at several patch sizes. The paper addresses a limitation it identifies in earlier pixel-space designs, which typically used two scales and sent information mainly from coarse representations to fine ones.
Its central question is whether adding scales and letting them exchange information can improve image generation.
- The middle scale helps
Adding an intermediate 4x4-patch stream improved FID, a score comparing generated-image distributions with reference-image statistics; lower scores mean a closer match. This was the 80-epoch ImageNet test, and the added stream also increased model size.
- Where capacity goes matters
With similar parameter counts, the intermediate-scale model had a lower FID, a score comparing generated-image distributions with reference-image statistics where lower means a closer match, than the coarse-only allocation.
Because counts were not identical and block allocation changed, this supports scale as a useful capacity choice in the tested setup.
- Two-way exchange is useful
The researchers also restricted information exchange in the late-anchored model. FID, a score comparing generated-image distributions with reference-image statistics where lower means a closer match, was 2.31 with unrestricted attention, 3.47 with the scale-causal mask, and 2.55 with the stricter mask.
This supports bidirectional exchange in that ImageNet setup. Longer-trained models and a text-conditioned version were evaluated separately, extending the reported evidence to other resolutions and text prompts.
- Results across tested settings
The longer-trained ImageNet results and text-prompt scores extend evaluation beyond the 80-epoch scale comparisons. The scale-scaling evidence itself remains limited to two through five scales in 80-epoch ImageNet tests.
Deep
- Original paper authors
- Ruibo Chen, Kaishen Wang, Tianyi Xiong, Chenxi Liu, Yanshuo Chen
- Submitted to arXiv
- License
- Not specified by source