A vision-language reward model learns from caption edits

Can a scorer's agreement with human preference tests tell us whether it will choose useful answers from a model's own candidates? The paper asks this because multimodal preference labels can be costly and disagree, and a high test score may not capture how a scorer behaves on those candidates.

The paper trains a frozen-model scorer on constructed comparisons, then tests how well it ranks human preferences and selects a model’s own candidates.

The method trains a small scoring head on a fixed vision-language model, using caption edits tied to the same image instead of human preference labels. The authors ask whether agreement with preference tests predicts how well a scorer chooses among a model's own candidates.

On a test of human-labeled response preferences, the 32B probe scored 0.724 pairwise accuracy versus GPT-4o's published 0.658.

But two probes with similar preference-test accuracy differed on a hallucination test, and a scorer trained on verified candidate captions reduced absent-object mentions against greedy selection.

The caption result measures absent COCO-object mentions, and CHAIR supplies both the training labels and the evaluation measure. It does not establish broader factuality or preserved informativeness.