Text-to-image models can learn without text-only pretraining

A key question for text-to-image research is whether models need internet-scale training on text alone to understand captions, or whether paired captions and images can teach that link directly.

The answer also affects system comparisons because text-only training can make up much of a system's total pretraining compute.

Halcyon learns caption-conditioned image generation by predicting text and image tokens in one sequence, without text-only language pretraining.

The authors train Halcyon from scratch on GPIC, using paired images and captions so the captions condition image generation. They compare generated images from 50,000 GPIC test captions using gFD, a measure of distance between generated-image and test-image distributions; lower is better.

Halcyon scores lower than selected systems that use pretrained text models, and the authors count language, image-generator and image-tokenizer training in total compute.

This shows that strong image generation is possible without text-only pretraining in these comparisons, while the differing systems leave the specific contribution of pretraining unresolved.

The main comparisons are between systems that differ in architecture, initialization, image representation and training procedure, so they do not isolate the effect of language pretraining.

The controlled tests cover GPIC and specific autoregressive models; the authors say results on other datasets and higher resolutions remain to be established.