A decoding method skips image processing when predictions agree
Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs · arXiv:2610.36671
In plain words
Diffusion vision-language models repeatedly process an image while gradually filling in a response. The paper introduces LOWN, a way to check whether the image is changing the next choices before paying to process it again.
On ten tests with LLaDA-V, LOWN kept average task scores close to the model’s usual decoding while using about a third of its estimated decoding compute. The savings vary: some inputs need the image much more often than others.
The research
When a vision-language model fills in a response over many steps, must it process the image each time? The image dominates the input, but some steps may make the same choice without it. The challenge is to identify those steps while still revisiting the image when it changes a decision.
LOWN checks text-only predictions against recent image-based ones, reusing agreements and revisiting the image when needed to guide response decoding.
The authors compared image-based and text-only predictions at each step along LLaDA-V’s standard decoding path. The image-based pass alone controlled the response, while the text-only pass served as a measurement.
Agreement was common, but image dependence varied across steps and tests, motivating a method that decides when to look at the image as decoding proceeds.
On ten tests, LOWN kept average task quality near the baseline while substantially reducing estimated decoding compute. A matched random schedule scored lower on three tests, supporting the value of LOWN’s agreement-guided choices in that comparison.
These results come from specified decoding tests, including a controlled video protocol that differs from VidLaDA’s released evaluation.
Savings also vary by input: when a document question needs the image repeatedly, LOWN may run a FULL forward on most steps, and its cost per response cannot be set in advance.
What the evidence shows
- Why process the image again?
Diffusion vision-language models refine a partly masked response over repeated steps. In LLaDA-V, visual tokens made up most of the input, while a forward pass without them cost much less than a full pass.
The authors asked whether the model needs to process the image at every step, or whether its current text and response state often already determine the next choices.
- Agreement varies by step
To measure image dependence, the authors compared image-conditioned and text-only predictions along LLaDA-V’s ordinary decoding path. The image-conditioned predictions alone determined which positions were revealed.
On ten task tests, both passes chose the same top prediction for 83.4% of the positions selected for revealing, on average. But the image changed at least one revealed choice on 22.8% of steps on average, with substantial variation across tests.
- A check guides two decisions
LOWN uses that variation to guide decoding. It keeps recent image-conditioned predictions, checks them against a text-only pass, and reuses the longest confidence-ordered run of agreeing predictions.
If too few agree, it processes the image again and refreshes its stored predictions. The agreement rule also determines how many response tokens to reveal, so image access and response progress adapt to the current input and step.
- Evidence and its reach
In standalone tests on LLaDA-V, LOWN retained 99.6% of average baseline task quality across ten tests while using 0.353 times the baseline’s estimated decoding compute. A random image-access schedule using a similar average number of full passes scored lower on three tests, which supports the value of LOWN’s selection rule within that smaller comparison.
Results on LaViDa and VidLaDA provide evidence on other model backbones, under their reported evaluation settings.
Deep
- Original paper authors
- Donghyeon Cho, Joowon Kim, Seungho Shin, Eunho Yang
- Submitted to arXiv
- License
- arXiv non-exclusive