SpecSlot separates image detail from global communication
Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers · arXiv:2610.40647
In plain words
Pixel-space image generators build pictures directly as pixels, but they often use each token both to represent a patch and to communicate across the image. More detail can therefore make global communication costly.
The paper introduces SpecSlot, a method that gives each image region several slots with different output jobs. Only the slot for broad image structure communicates across regions; detail slots work locally.
On ImageNet-512, SpecSlot used less forward computation than JiT-B/16, though its average FID image-quality score was slightly worse.
The research
When pixel-space image models represent finer detail, their spatial-patch designs can also require more tokens in costly global communication. Can a model allocate more roles to detail without expanding the global sequence that carries information across an image?
SpecSlot assigns image detail to local slots while routing one broad-structure slot per region globally, separating detail roles from global communication.
On class-conditioned ImageNet-512, the authors compare SpecSlot with JiT-B/16 after aligned 120-epoch training. They sample images under a shared evaluation protocol.
Across five trained seeds, SpecSlot’s mean FID score, computed from 50,000 images per seed, was slightly worse, while its conditional forward compute was lower. This supports a quality-compute trade-off in this setting, without identifying which design difference explains it.
These results cover the paper’s tested pixel-space image-generation settings. The ImageNet-512 comparisons do not isolate which architectural differences explain SpecSlot’s results or the larger margin over the compute-matched control.
What the evidence shows
- The detail and communication trade-off
SpecSlot is a pixel-space image-generation method that assigns different output roles to slots within each image region. The paper asks whether a model can represent finer detail without sending every detail token through costly global communication.
This question matters because, in the spatial-patch designs discussed, more detail means more tokens participating in that communication.
- Give slots different roles
SpecSlot reads input at spatial anchors and allocates each anchor’s output across indexed slots. Each slot owns a separate group of frequency coefficients, from broad structure to fine detail, and the slot states also produce a local pixel correction.
Only the broad-structure slot at each anchor enters global attention; all slots interact locally. The design separates the number of detail roles from the number of global image carriers.
- Check the carrier choice
A post-hoc probe measured when each frequency group’s signal became stronger than its noise in trained checkpoints. In all three reported allocations, the lowest-frequency group reached that threshold earlier than the higher-frequency groups.
This supports the choice to route that group globally, but the probe measures signal timing, not image quality or a learned routing decision.
- Read the quality and cost results
For class-conditioned ImageNet-512 generation, the authors compared SpecSlot with JiT-B/16 and a compute-matched JiT control. SpecSlot used fewer conditional forward operations than JiT-B/16 but had a slightly worse mean generated-image quality score; against the control, it had a better mean score at slightly lower forward cost. The control’s design differs, so neither comparison isolates one architectural cause.
Deep
- Original paper authors
- Jiongfan Yang, Weichen Fan, Haiwen Diao, Zhongang Cai, Ziwei Liu
- Submitted to arXiv
- License
- CC BY 4.0