In plain words
Text-to-image models can have trouble placing objects where coordinates in a prompt specify, even when their text encoders can represent spatial information.
This paper introduces a way to train existing flow-matching models further by changing how often they see different noise levels, without changing their architecture.
On a test made from colored rectangles, a shifted training schedule scored much higher on layout and color overlap than a centered schedule. The result is limited to that synthetic test, and strong shifts can reduce color or texture precision.
The research
If an image generator can represent spatial information in its text encoder, why might coordinates in a prompt still fail to place objects correctly? The paper tests whether changing how the model is trained can help it use those coordinates without adding structural parts to the model.
The method fine-tunes image generators by showing them higher-noise stages more often, helping them learn to place objects from prompt coordinates.
The authors test this method on synthetic prompts describing colored rectangles, where each generated image can be compared with a pixel-level reference. They vary the mean of the noise-level distribution while holding its standard deviation fixed.
On the 256x256 test, the layout score is 25.9 with a centered distribution and 80.8 with mean 1.0; the VAE reconstruction reference is 83.6. The score averages overlap between generated and reference color masks across colors and color-matching thresholds.
The result supports the view that shifting training toward higher noise can help the model follow coordinates in this setting. Further tests show the best shift varies with resolution, and a detector-based measure also improves on selected COCO images.
The results show that changing the training noise distribution can improve layout scores in these tests, but they do not establish that every model or image setting will benefit.
Stronger shifts can reduce texture or color precision, and the real-image result relies on a detector as a proxy for layout accuracy.
Analysis
Deep
- Original paper authors
- Martin Thißen, Elke Hergenröther
- Submitted to arXiv
- License
- CC BY 4.0