Match visual compression to computation and cache memory
An image question may seem easy after most visual tokens are removed, yet a small word or an object's location can still be lost. A separate constraint appears during answer generation, when cached visual information occupies memory.
The first useful step is to identify the constrained stage. Reducing visual tokens changes the representation the model processes; evicting cached visual information changes what later generated words can draw on. The studies test these choices with different models and questions.
What you will understand
Is the constraint the number of visual tokens processed or the visual information stored for later generation?
When does an image question call for distinct visual details, and when might ordered merging be useful?
Does the chosen benchmark score the part of generation that cache eviction can change?
Name the constrained resource
Imagine an assistant that must both read text in a picture and point to a named object. Its operator wants to send fewer visual tokens into the language model.
A visual token is a piece of the image represented for the model to process. Keeping fewer pieces may help with the processing budget, but a score on ordinary image questions may conceal trouble with reading or locating small details. The Bifocal study found that tension across its tested question types.
The word “budget” can refer to different resources. Bifocal reduces the visual-token sequence before and inside a language model.
LapMerge also produces a shorter sequence, but combines ordered inputs to meet a chosen output count. TwinPrune addresses that processing stage and a later storage decision: how many visual entries remain in the model's cache while it generates an answer. A shorter input sequence and a smaller cache therefore describe different interventions.
The timing matters. The model first processes the image and question together, a stage called prefill.
It then generates answer tokens one after another, using stored visual information during decoding. If the concern is work done while processing the input, reducing or combining visual tokens is relevant.
If the concern is memory held during later words, cache eviction is relevant. The shared question is what evidence each decision leaves available, and which test can reveal a loss.
Where the lesson goes next
Keep complementary visual evidence
Use order to combine tokens
Test cache eviction during generation
Apply the choice within its tested limits