Multimodal models need vision capacity, but not a separate encoder
Unbundling Multimodal Vision Encoding · arXiv:2611.17334
In plain words
A multimodal model handles both pictures and language. Researchers tested whether it needs a separately trained image-reading component, or whether the important ingredient is extra processing used only for pictures.
They trained models from scratch with shared data and compute. In their tests, models with picture-specific processing did better on visual questions than a model without it, while putting that processing inside the shared model worked about as well as a separate encoder. The result applies to the tested models and tasks.
The research
For a model trained on the same data and compute, the choice is whether to spend resources on a separate visual training stage or preserve capacity that processes images specifically.
These comparisons shift the design question: the tested visual gains tracked dedicated vision processing, while its location before or within the shared model mattered less.
A controlled study separates visual pretraining, vision-only capacity, and its placement, finding that dedicated vision processing helps tested visual-question scores without requiring a separate encoder.
The researchers trained models from scratch on common data, matching total compute across training stages. They measured average scores on visual question answering tests, which ask questions about images, and on language tests.
A jointly trained vision encoder performed comparably to separately pretrained encoders, while removing dedicated vision processing lowered the visual-question average.
That pattern points to capacity reserved for images, rather than a required separate pretraining stage. A second comparison found that this capacity could sit inside the shared model and remain comparable to a separate encoder.
The tests used dense backbones up to 6.4 billion parameters and 224x 224 images during multimodal training; the encoders saw the same images as that training. They leave open what additional pretraining data, higher-resolution images, or other model and data families would change.
What the evidence shows
- Separate bundled choices
The controlled multimodal vision-encoding study is a protocol for separating design choices that published-model comparisons often mix together.
Those models can differ in data, training stages, compute, and architecture, so a performance gap alone cannot show whether it came from pretraining, dedicated vision capacity, or where that capacity operates.
- Separate pretraining was optional
The study's visual question answering tests ask models to answer questions about images, and the reported score averages categories across the test suite. With total training compute matched, the encoder trained jointly from scratch performed comparably to encoders that received separate visual pretraining and were then updated. In this setup, a separate pretraining stage was not necessary for comparable test averages.
- Dedicated capacity matters
Under matched lifetime compute, the scratch-encoder model averaged 44.4% across visual question answering tests, versus 39.0% for the model without a dedicated vision encoder.
This comparison points to a visual-task gap when dedicated vision processing is removed, though the models also differ in how image patches enter the backbone.
- Shared parameters did not close the gap
A parameter-matching control added shared parameters to the encoder-free model. Its visual-question average was 38.6%, compared with 44.4% for the scratch encoder.
Adding parameters usable by both text and image tokens did not close the gap in this control, supporting a role for capacity reserved for vision under the tested compute conditions.
- Vision capacity can move inside
The study also placed vision-only modules inside the shared model. At the longest reported budget for the smaller tested backbone, this in-fusion design averaged 49.2% on visual question answering, versus 48.9% for a separate encoder.
The close averages support the conclusion that dedicated processing need not sit in a separate module before text and image processing meet.
- The pattern held with scale
Across the tested backbone-size ladder, fitted scaling laws estimated the compute needed to reach the same visual-question average. The encoder-free model needed 2.6 to 3.1 times the encoder's compute, while in-fusion needed about the same.
This extends the capacity and placement pattern across the study's sizes, but remains a fitted result for its tested recipes.
Deep
- Original paper authors
- Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis
- Submitted to arXiv
- License
- CC BY 4.0