Video scores shift with setup, so Meta-Video records the settings
In plain words
The paper introduces Meta-Video, a compact set of questions about videos for comparing video-language models. The authors remove questions that several tested models can answer even when the video is replaced by black frames, then select questions from five existing tests.
Meta-Video largely keeps the model rankings seen on those source tests while using one-fifth the inference compute for a full sweep from 4 to 128 frames. The result applies to that sweep; the paper also shows that evaluation choices can change scores.
The research
When the same model gets different scores on the same video test, how can researchers tell whether a reported gain reflects the model or the way it was evaluated? On VideoMME, published accuracy for one checkpoint spans 19.5 percentage points, making that distinction difficult.
The authors introduce Meta-Video and a reference protocol that records video-evaluation settings so scores can be interpreted and repeated.
They remove questions that three models can answer with black frames, then sample 1,000 questions from five existing tests. Meta-Video largely preserves source-test model rankings while using one-fifth the inference compute for a full 4-to-128-frame sweep.
Separate evaluations show why settings matter: changing the backend or frame count can change scores, and frame count can change which model ranks higher. The protocol records settings alongside scores so such comparisons can be interpreted.
The reported comparisons cover five open models, seven source tests and Meta-Video under specified evaluation settings. They do not establish that the same rankings or compute savings hold for other models, tests or setups.
Analysis
Deep
- Original paper authors
- Soumya Shamarao Jahagirdar, Anna Kukleva, Vishaal Udandarao, Samuel Thomas, Brian Kingsbury
- Submitted to arXiv
- License
- CC BY 4.0