Smaller vision models can outperform larger ones on specific tasks

When a vision-language model gets better overall, can it still become less accurate on a particular kind of visual reasoning? Aggregate scores combine categories, so they may hide questions where a smaller model answers correctly and a larger one does not.

The study compares smaller and larger models on shared questions to find category-specific declines that broad test scores may conceal.

To look for category-specific declines in ordinary tests, the authors compare models within each of four open-weight families on shared questions. For each test category, they count disagreements favoring the smaller model and subtract those favoring the larger one. A positive balance means an advantage among disagreements, not on every question.

They find 375 qualifying category-and-model-pair cases among 10,434 eligible comparisons. BLINK tests visual perception across multiple images; on its Jigsaw category, Qwen3-VL 4B was 24 percentage points more accurate than Qwen3-VL 30B-A3B on the same 150 questions. The example shows how category-level declines can be missed by broad scores.

Because this is an observational comparison of released model variants, which may differ in more than size, it cannot show that parameter count caused the declines. The study also covers open-weight models only.