A training method helps video models use visual evidence
Watching Itself Watch: Self-Auditing Visual Reliance for Video Reasoning · arXiv:2611.11949
In plain words
A video question may depend on a brief moment that a model’s fluent explanation can overlook. VIOS is a way to train video question-answering models to pay more attention to such moments.
The model drafts an answer, then learns from a copy of itself that also sees the correct answer. VIOS gives extra weight to words where the teacher’s next-word prediction changes more when the video is included than the student’s prediction does.
Across 12 video-understanding tests, average scores rose for all three tested models compared with their starting versions.
The research
A fluent answer can miss the brief visual moment that determines whether it is right. Knowing whether training can help video question-answering models rely more on what they see changes what counts as useful supervision: a whole-answer score may miss where the model needs to attend. VIOS tests a way to target those positions.
VIOS trains video question-answering models to focus on answer positions where an answer-informed teacher relies on video more than the student does.
The method has a student answer each video question, then compares its next-word predictions with those of a teacher that also sees the correct answer.
By measuring how each prediction changes when the video is removed, VIOS gives more learning weight to positions where the teacher relies on video more than the student.
Across 12 video-understanding tests, average scores increased over each model’s corresponding starting checkpoint for all three tested backbones. The gains support the method on this evaluation set.
The results cover three tested model backbones, trained on one video-question-answering corpus, and 12 video-understanding tests. The training examples need correct final answers, and the method encourages video use without ensuring that a claim is tied to a particular frame or region.
What the evidence shows
- Locate missed visual evidence
Visually-Aware On-Policy Self-Distillation (VIOS) is a post-training method for video question-answering models. It addresses a mismatch: an answer may depend on a few visual moments, while a score for the whole answer does not identify where the model failed to use them.
VIOS instead uses one student-generated answer to locate positions where a teacher’s predictions depend more on the video than the student’s do.
- Scores rose across three models
Because each VIOS-trained model is compared with its own starting checkpoint, the result asks whether post-training helps within each model family. Higher averages across the same 12 video tests show gains for all three tested models, though averages do not imply improvement on every test.
- Read the averages in scope
The comparison is informative because all three model-specific pairs start from the corresponding base checkpoint, and the reported measure averages scores over the same 12 video tests. The evaluation covers tests of general understanding, knowledge, temporal and spatial reasoning, and long videos.
This supports a conclusion about those models and tests, rather than a guarantee for other video tasks or model sizes.
- A bounded training-cost comparison
A separate Qwen2.5-VL-SFT comparison used Video-R1’s per-step training configuration. VIOS used over 9 times fewer steps, 60 times fewer total training FLOPs and 111 times fewer generated tokens than Video-R1.
This is a training-cost comparison between different procedures in that setup; it does not show that a single component caused the differences or that the same savings apply to all three models.
- The added objective contributes
This comparison isolates the contrastive objective within the Qwen2.5-VL-SFT setup, where distillation and the visual-reliance gate are already present. The higher average across 12 video tests after adding it supports a contribution in this setup, but does not establish the same increment for other backbones.
VIOS still needs correct answers in its training examples, and encouraging video use does not ensure grounding in a particular frame or region.
Deep
- Original paper authors
- Rohan Dugad, Mi Luo, Kristen Grauman
- Submitted to arXiv
- License
- CC BY 4.0