Why language models can overstate their progress
In plain words
This paper studies why a language model’s score can exaggerate the progress it makes by rewriting its instructions. The authors model how a loop chooses the best-scoring rewrite, then compare those scores with results on separate examples.
In tested loops, the reported score was much further above the separate-example accuracy when the loop used a small set of examples. That gap narrowed with a larger set, though real gains did not rise in every tested task.
The research
When a language model repeatedly rewrites an instruction, the score used to choose each rewrite can also become its progress report. How much of an apparent gain reflects a genuinely better instruction, and how much comes from repeatedly selecting the luckiest score?
The paper models how selecting the best-scoring rewrite can inflate reported progress, then tests that effect in instruction-editing loops.
In a pre-registered study, greedy loops reused their selection examples but received feedback from a separate pool. Researchers compared selection scores with accuracy on a separate 600-example gold set across four settings. The reported gap was smaller with 256 selection examples than with 16.
Actual held-out gains rose with set size in both TREC settings, but not in Banking77 or GSM8K. Banking77’s mean gains were -0.6, 0.0, and -0.5 points with 16, 64, and 256 examples; no GSM8K arm gained more than 3.1 points.
This supports the concern that selection scores can overstate progress, while showing that a smaller reporting gap does not guarantee greater real gains.
The experiments cover instruction editing with tested models and scoring setups. Larger selection sets did not increase held-out gains in every task, and the results do not establish how other systems or tasks would behave.
Analysis
Deep
- Authors
- Litao Hu, Yutong Tang
- License
- CC BY 4.0