EN
A two-stage method separates visual learning from rendering
Know, then Draw: A New Learning Paradigm for Visual Generation · arXiv:2610.37626
In plain words
Making an image from a sentence involves understanding what the sentence describes and drawing the details. This paper introduces a method that gives those jobs to two training stages: first a model learns visual codes from text and images, then a renderer uses the codes to make pixels.
In the tested settings, this route scored higher on image-generation tests than the evaluated route that began with latent-diffusion training. The result applies to those comparisons, not every way of training an image model.
The research
Text-to-image generation requires a model to learn what a prompt depicts and to render that knowledge as detailed pixels. The paper asks whether one training approach needs to do both jobs, or whether learning visual knowledge and drawing images can be assigned to different stages.
The method uses autoregressive pretraining to learn visual codes, then diffusion post-training to turn those codes into detailed images.
The researchers compared autoregressive pretraining followed by pixel diffusion with latent-diffusion pretraining followed by pixel diffusion.
On GenEval, which measures how well images match prompt details across six categories, and DPG-Bench, which measures text-image alignment, the autoregressive route scored higher under matched image exposure and about 400 million final-generator parameters.
That comparison supports the route in the tested setting. It does not isolate pretraining objective as the only cause, since the routes also differ in representations and architecture details.
The comparison supports the tested training routes at the reported model size and image exposure. The routes also differ in representations and architecture details, so it does not isolate pretraining objective as the only cause.
What the evidence shows
- Assign learning and drawing
The paper proposes a two-stage method for text-to-image generation: an autoregressive model learns visual knowledge by predicting codes from text and images, then a diffusion renderer uses the learned representations to make pixels.
This division addresses the paper's question of whether the objective best suited to learning visual information is also the one best suited to rendering detailed images.
- Compare final image quality
To test the training routes, researchers compared autoregressive pretraining followed by pixel diffusion with latent-diffusion pretraining followed by the same rendering architecture and post-training protocol. The routes had about 400 million final-generator parameters and matched cumulative image exposure, though their representations and some architecture details differed.
GenEval checks how well images follow prompt details, while DPG-Bench measures text-image alignment; the autoregressive route scored higher on both in this comparison.
- Examine what codes preserve
The authors then asked what visual codes the learning stage should predict. With generator training and evaluation held fixed across three tokenizer conditions, they found that strong reconstruction or standalone image-generation scores did not reliably indicate strong final rendering.
A semantic tokenizer trained through language modeling also supported effective rendering despite having no image decoder or reconstruction supervision. This suggests the codes can focus on semantic information when a separate stage handles drawing.
- Scale and test the recipe
Scaling fits for the two stages suggested different ways to balance model size and training data. A comparison of fitted rendering-loss floors also favored the post-trained pipeline over an independently tuned pixel-diffusion baseline trained from scratch, but these are extrapolated fits for whole pipelines, not an isolated test of pretrained weights.
The authors used the scaling analysis to guide a roughly 3-billion-parameter model, which achieved competitive image-generation scores at the stated compute budget under an evaluation protocol that rewrote and selected prompts.
Deep
- Original paper authors
- Yucheng Mao, Chongjian Ge, Zhuowen Tu, Hao Tan, Hanwen Jiang
- Submitted to arXiv
- License
- CC BY 4.0