A small set of heads helps multimodal models find evidence
A Broader Look into MLLM's Retrieval Heads · arXiv:2610.32411
![Retrieval heads are sparse in omni-modal models. (a,b) Calibrated scores (Equation 2) under audio evidence for two representative models, normalized to [0,1] per model; rows are layers and columns…](https://arxiv.org/html/2610.32411v1/fig_sparsity.png)
In plain words
A model answering questions about a long text, image, or video must find the small part that contains the answer. This paper studies whether a small group of attention heads helps multimodal models locate that evidence, and whether those heads can improve document search.
The authors found that many of the same heads respond when a six-digit answer in a video is spoken or shown on screen. They also trained those heads to rank document pages, improving page retrieval on one test; that result does not show broad improvement on other tasks.
The research
When an answer is buried in a long multimodal input, does a model use the same parts of its attention system to find it across different formats? And can identifying those parts help improve document retrieval?
The paper tests both questions, matching spoken and displayed evidence before using the identified heads in a ranking method.
The authors identify shared retrieval heads in multimodal models, then train their attention to rank document pages more effectively.
The authors identify retrieval heads by measuring how much attention they direct from a question to annotated evidence, relative to an uninformative question.
They apply this method across text, image-text, and audio-visual tasks, then compare paired videos where the same answer is spoken or displayed.
Most top-ranked heads overlap across the paired conditions, and removing selected heads harms held-out retrieval beyond a matched random control.
The authors use this attention as a trainable page-ranking signal: on MMDocIR, training improves Qwen3-VL-8B’s micro page Recall@1 over its untrained scorer under the same evaluation conditions.
The audio-visual tests use constructed videos with a six-digit answer spoken or displayed, and the document-ranking gain is measured on MMDocIR. Ranking training changed scores by less than one point on four other tests, so these results do not establish broad transfer.
What the evidence shows
- Finding evidence in long inputs
The paper analyzes retrieval heads, a sparse set of attention heads that direct question-related information toward evidence in an input, and introduces a method to train their attention to rank document pages.
The question is whether these heads continue to help when models process audio and video, and whether understanding them can guide a concrete improvement.
- A shared way to find heads
To identify candidate heads, the authors compare attention from question tokens to annotated evidence with attention measured using an uninformative question. They average that difference across examples and rank heads.
Applying the same method across text, interleaved image-text, and audio-visual tasks makes it possible to compare evidence use across 18 model checkpoints.
- The same answer, two formats
The paired audio-visual test holds the video, question, answer, and answer position fixed while changing whether the six-digit answer is spoken or displayed. Across eight omni-modal models, 69% to 83% of the top-ranked heads are selected in both conditions; independent random sets would share fewer than three.
This overlap supports reuse of many of the same heads for the matched answer, within this constructed task.
- Testing whether heads matter
The authors then test whether identified heads are needed by zeroing their outputs and comparing held-out retrieval accuracy with both an unablated model and a layer-matched random-head control. Removing the top 5% eliminates audio retrieval in all four tested omni-modal models and removes most visual retrieval.
Across eight broader tasks in two models, ablating heads shared across audio and visual evidence causes score drops beyond matched random removal on all eight tasks. A separate control uses equally-sized heads selected on natural-image retrieval; the audio-visual-shared-head drop is larger in 15 of 16 model–task pairs.
- Turning attention into rankings
Finally, the authors use attention from selected heads to rank candidate document pages and train that signal to favor relevant pages over hard negatives. On MMDocIR, Qwen3-VL-8B’s page Recall@1 rises by 6.03 percentage points over its untrained scorer under the same preprocessing and candidate windows. Changes on four other tests remain under one point, bounding the evidence for transfer.