Frame choice had bounded gains in a long-video audit
Where Do Reported Long-Video QA Gains Come From? A Controlled Audit of Frame Selection, Evidence Injection, and Expert Routing · arXiv:2611.07329
In plain words
A system answering questions about a long-video cannot show its model every frame, and some answers depend on speech or text that the pictures do not contain. VideoRouteAudit is a training-free harness for checking frame choice, added text and the answering model.
On 900 questions about long videos, its tested frame-selection cascade scored close to evenly spaced frame sampling: the measured difference was small enough to rule out gains of five percentage points or more, but not smaller gains. Transcript effects also changed with the video and prompt wording.
The research
A long-video system has to spend a limited frame budget, and evidence in speech or text may matter as much as what appears in the images.
This audit changes what a small measured gain can justify: on its full long-video test, it ruled out large selection gains but left smaller ones open, while showing that added text and the answering model can shift results.
VideoRouteAudit measures frame selection, added text and answering-model choice, finding that selection gains were bounded while text effects depended on setup.
The audit compared a cascade selecting eight frames with uniform sampling while using the same answering model on all 900 Video-MME long questions.
Exact option-letter accuracy was 1.2 percentage points higher with the cascade, and the five-point equivalence test ruled out gains that large in this setup, though smaller gains remained possible.
Resampling 200-question subsets from those same outcomes reversed the full-set difference’s sign in 35.8% of draws. That instability supports caution about small-sample claims of modest selection gains; it does not show that selection has no effect.
The results cover the tested datasets, models, frame budget and configurations. The soundtrack controls used one expert on a type-stratified subset, while the transcript-placement analyses were post hoc and unadjusted.
What the evidence shows
- What the audit measures
VideoRouteAudit is a training-free audit harness that measures frame selection, added text and answering-model choice in long-video question answering. The central question is whether changing frame selection meaningfully improves answers when the other relevant conditions are matched.
Video-MME long tests question answering about long videos; its multiple-choice accuracy counts answers whose parsed option letter exactly matches the correct choice.
- A bounded selection effect
On the full set of 900 Video-MME long questions, using eight frames and Qwen2.5-VL-7B, the cascade scored 1.2 percentage points above uniform sampling. The comparison passed an equivalence test within a five-point margin, so gains of that size were ruled out for this tested setup.
It did not pass the tighter three-point margin, leaving smaller gains unresolved. The result bounds the tested selection effect rather than showing that frame selection never matters.
- Small samples shift the estimate
Because these estimates come from smaller samples of the same cached outcomes, their direction can vary with which questions are included. This makes small-sample comparisons poor evidence for a modest selection effect or a precise null in this evaluation.
- Transcript effects need context
The reruns show that transcript effects varied with both the video domain and where the text appeared in the prompt. The published anomaly results labeled “+ASR” cannot establish an injection effect because those runs added no transcript text.
- The answering model matters
The MUSIC-AVQA comparison changes the answering model, so its gain cannot be attributed to audio alone. Soundtrack controls show that the expert uses audio, but most of its advantage over the visual-only model remained when it received silence.
Deep
- Original paper authors
- Shih-Yao Lin
- Submitted to arXiv
- License
- arXiv non-exclusive