Read the evaluation protocol before trusting a benchmark score

Imagine seeing different accuracy scores for the same video model on the same benchmark. The model weights may be unchanged, yet the evaluator can choose different frames, process them differently, or score the response by a different rule.

Across video questions, Taiwan knowledge questions, and dependency repairs, these studies show why a result needs its protocol beside it. A compact test can make a comparison less costly in its tested setting, while its task coverage still limits the claim. For a repair, success also depends on which checks the verifier runs.

A researcher comparing video models finds a troubling case: the same Qwen2.5-VL-7B checkpoint has a reported VideoMME accuracy range of 19.5 percentage points across papers. That figure is the span between reported scores for one fixed model on one video question benchmark, not a measured improvement in the model.

If a claimed advance sits within such variation, the score alone does not show whether the model or its evaluation setup produced the difference.

The video study points to choices a benchmark number can conceal. A model may receive different sampled frames from the same clip; a frame limit changes how much of the clip it can inspect.

The software reading and preparing video can also affect the input. A reader deciding whether to trust a model comparison therefore needs to know what reached each model, as well as the question it was asked.

That problem extends beyond video. A language model can give a response that one answer reader accepts and another mishandles.

A coding assistant can leave a project building while failing to remove its targeted vulnerability. These are different tasks, but they pose one shared question: what did the evaluation actually supply, count, and check? The following comparisons answer parts of that question within their own tested settings.