Bifocal preserves image detail and context while trimming visual input
In plain words
An image-reading model turns a picture into many small pieces it can inspect. The paper introduces Bifocal, a method that cuts down those visual pieces without extra training: it keeps some that match the question, some that represent the image more broadly, and small summaries of what remains, then removes more pieces inside the model.
In tests on three model families, Bifocal kept more than 90% of each unpruned model's performance in every tested ability group while using roughly a quarter of the visual pieces.
The research
Can a model process far fewer image tokens without losing the fine visual evidence needed to read text or locate objects?
Common visual-question scores did not answer that: the paper's broader tests found that methods that looked strong on general questions could lose much more on fine-grained tasks.
Bifocal combines question-focused patches, image-selected patches, and pooled image summaries, then prunes again inside the model using text attention and local token density.
At roughly one quarter of the visual tokens, image-ranked patches favored text-reading and document tasks, while question-ranked patches favored object location. An even split tracked the better single route, suggesting complementary cues.
Bifocal combines both patch types with pooled summaries, then prunes inside the model. Across five ability groups and three model families, it alone kept over 90% of each unpruned model's performance in every group.
These results cover the tested ability groups and model families. On LLaVA-NeXT, images produce different numbers of visual tokens, so its average token budget is not a fixed fraction of each image's input.
Analysis
Deep
- Original paper authors
- Peiran Yin, Hao Wu, Zihao Yang, Fengyu Zhou, Yijuan Liang
- Submitted to arXiv
- License
- CC BY 4.0