Many tested object-attribute errors show detectable visual evidence

When a model answers an image question incorrectly, can we tell whether it lacked the needed visual information or failed to use information it had? The final answer alone cannot distinguish those causes, yet they imply different sources of failure.

The authors introduce VELens, a probe that estimates whether wrong image answers reflect missing visual evidence or evidence the model failed to use.

The authors introduce VELens, a probe designed to address whether wrong answers reflect missing visual evidence or evidence a model has encoded but failed to use. It checks internal representations for both the target object and its attribute.

On held-out object-attribute instances, the combined probe distinguishes evidence-present from evidence-absent labels. Its ROC-AUC score measures how well it makes this distinction across decision thresholds: 0.95 for LLaVA and 0.91 for Qwen.

Using it, the authors estimate that most tested errors occur despite detected evidence, suggesting that using visual information is a major bottleneck in this setting.

These estimates rely on a probe that can miss existing evidence or detect spurious cues, and the lower bounds rely on the paper’s probe-dominance assumption. The results cover two models and a limited set of object-attribute questions, not visual questions generally.