A frozen multimodal model can rebalance conflicting cues
Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs · arXiv:2610.27103
In plain words
Audio-visual language models answer questions using words, pictures, and sound. Sometimes one cue overpowers another: a model might answer from a picture even when the question asks what can be heard.
Omni-Woodpecker checks separate views of the same input, then edits a frozen model’s internal information to protect useful evidence and reduce a competing cue before the answer.
On two tests of audio-visual hallucinations, it improved answer accuracy over the unedited model on both of the two models tested.
The research
When an audio-visual model answers from the wrong cue, is the relevant evidence absent, or did another cue overpower evidence the model had already encoded? The distinction matters because a correction that only changes the final answer may leave the internal conflict untouched. The authors examine whether rebalancing that internal representation can improve answers.
OWP compares separate views of a frozen audio-visual model, then edits its internal representation to protect query-relevant evidence before answering.
OWP compares a frozen model’s answers to the full input and separate audio, video, and text views to detect when the needed evidence conflicts with the answer. It then edits the internal representation before answering, protecting evidence and reducing the competing component.
On CMM and AVHBench, tests of cross-modal hallucination and audio-visual matching, OWP improved native-format accuracy over the unedited Qwen2.5-Omni-7B model and the MAD method, which adjusts answers using modality cues. This supports the approach on those tests and that model.
These results cover cross-modal hallucinations on the tested benchmarks and two open-weight models. OWP needs access to model internals and cannot restore evidence the model failed to perceive in the relevant modality.
What the evidence shows
- When cues compete
Omni-Woodpecker (OWP) is a training-free method for correcting cross-modal hallucinations in frozen audio-visual language models. The authors frame the problem as a conflict: a model may encode evidence in the modality a question asks about, yet answer according to another modality or a language-based expectation.
OWP addresses this particular account of hallucination by editing internal representations before the model produces an answer.
- Finding the conflict
To locate a conflict, OWP compares the full input with views containing the question and just audio, just video, or text alone. CMM tests cases where audio, visual, or language cues conflict; its accuracy measures correct answers under its test format.
AVHBench tests audio-visual hallucination and also includes questions about whether sound and video match. In covered CMM and directional AVHBench rows with complete view records, OWP flagged 25.8% of inputs, changed 16.9% of answers, and left the rest unchanged. The matching task is excluded from those rates because it has no single target stream.
- Testing the two streams
A shared binary yes/no probe on Qwen2.5-Omni-7B separates the roles of the streams. On the same rows, using only the benchmark-labeled target stream scored 84.28% accuracy, while using only the competing stream scored 56.85%; the full-input model scored 82.06%.
OWP edited the full input and scored 84.91%. This comparison supports the idea that a competing stream can harm an answer even when target evidence is useful, while noting that OWP infers the target stream and the stream-only controls use benchmark labels.
- Measuring gains and bias
On the tests’ native answer formats, OWP scored 86.1% on CMM and 83.7% on AVHBench with Qwen2.5-Omni-7B, beating the unedited model and MAD, which adjusts answers using modality cues. A control shifted the model’s yes-minus-no score toward “No” to match OWP’s yes-answer rate; this raised accuracy by 0.60 percentage points, compared with OWP’s 2.85-point gain on the shared probe.
The control suggests that this uniform shift does not account for the full gain. OWP also improved accuracy on VideoLLaMA2-AV, one other evaluated model.
Deep
- Original paper authors
- Zijian Liu, Chaoyou Fu, Yan Yang, Caifeng Shan, Ran He
- Submitted to arXiv
- License
- arXiv non-exclusive