Suppressing image-to-text feedback improves generation
En termes simples
This paper studies a loop in some image generators: the unfinished picture can send information back into the text instructions used for the next update.
It introduces Semantic Feedback Guidance, a training-free method that weakens this image-to-text feedback and strengthens text conditioning while generating.
In paired tests, borrowing the feedback from a successful run rescued 67.5% of failed runs, compared with 1.3% for a random message of the same size. A direct version made the change without extra model evaluations.
La recherche
When an image generator starts missing an object, should its internal feedback help recover the prompt or reinforce the incomplete picture? The answer matters because the same loop that lets image and text streams interact can also carry prompt-conflicting content forward.
This paper asks whether changing that loop at sampling time can improve outcomes without retraining the model.
The authors trace failures to image-to-text feedback, then introduce SFG and iSFG to suppress it during early sampling, with iSFG requiring no extra model evaluations.
The researchers first transplanted the image-to-text message between same-prompt success and failure pairs while keeping the recipient prompt, seed, sampler, other activations, and 1024x1024 setup fixed. A successful donor rescued 270 of 400 failed recipients, while a norm-matched random message rescued 5 of 400; a failed donor also disrupted 242 of 400 successes.
That contrast identifies message content, not just its size, as relevant to the trajectory. The resulting SFG method guides away from a branch with stronger image-to-text feedback; iSFG applies the suppression directly.
On FLUX, GenEval, a test of whether images contain requested objects and relationships, rose from 0.6661 to 0.7479 with SFG, while iSFG raised SD3.5M’s HPSv2.1 preference score from 31.65 to 32.98 without extra evaluations.
The evidence is bounded to the tested paired trajectories and evaluation sets, so aggregate gains do not guarantee every prompt or output.
The best guidance window also depends on the task: for HunyuanVideo text-to-video, seven of 25 steps scored 84.25 versus 83.09 for all 25, using 32 rather than 50 model evaluations.
Analyse
Détails
- Licence
- arXiv non-exclusive