A projector can align vision and language without captions

Projector alignment is a standard first step in training multimodal language models, yet studies disagree about whether it is needed and how much paired image-caption data it takes.

What does that step actually contribute: detailed links between images and captions, or a more basic compatibility between the image and language features?

PORTAL builds a vision-to-language projector from separate feature statistics, avoiding paired captions and alignment optimization while staying near trained baselines in tested models.

The authors tested whether alignment needs detailed image-caption learning by reducing the paired data and blending projectors trained on different datasets. Most gains appeared with a small data fraction, and cross-dataset blends remained effective when the projectors shared an initialization.

They then built PORTAL from separate feature statistics. After identical instruction tuning, it averaged close to the trained projector across seven tested model combinations, supporting coarse alignment as a useful account.

These findings concern frozen vision and language backbones, single-image inputs, and the tested projector designs and model combinations. The authors describe coarse geometric conditioning as the simplest explanation consistent with their evidence, not the only possible mechanism.