Why uneven question difficulty can skew test scores
Your Benchmark Is an Empirical Measure Over Difficulty · arXiv:2610.29927
In plain words
A test’s average score depends on which questions it includes. If most questions are easy, the score may not reflect performance on harder questions; if no questions cover a difficulty range, improvement there may go unnoticed.
The authors develop a way to measure those patterns and a method for adding questions where coverage is sparse. In their analysis, both uneven coverage and large gaps appeared in many of the tests they examined, and targeted additions reduced gaps in a smaller group of tests.
The research
When researchers average results to compare models or track training, does that score reflect the difficulty mix they care about? And can it show improvement in parts of that mix that have few or no questions? The paper examines whether the questions themselves limit what these scores can tell us.
The authors model tests by question difficulty, derive mismatch and gap diagnostics, and introduce BenchFill to add questions where coverage is sparse.
The authors estimate question difficulty from model responses, then measure how each test’s difficulty distribution differs from a same-size uniform reference and how large its biggest uncovered interval is.
Among 25 tests, 10 crossed the mismatch threshold and eight had a largest gap of at least 0.10 on the normalized scale.
Their theory connects mismatch to possible distortion in model comparisons and the largest gap to whether localized improvement can be detected. The empirical counts show these conditions occur in the analyzed tests, while BenchFill tests whether targeted additions can reduce gaps.
The theoretical guarantees depend on their stated conditions: score distortion assumes model performance differences change smoothly with difficulty, and the detection result concerns the specified test for interval-wide improvements.
The empirical analysis estimates question difficulty from model responses, and the framework is primarily a post-hoc way to assess existing tests.
What the evidence shows
- What an average can hide
The authors introduce a framework for analyzing tests through the difficulty of their questions. The question is whether an average score reflects the difficulty mix that matters for comparing models, and whether it registers improvement in regions the test barely covers.
The framework treats the questions’ difficulty distribution as part of what a test measures, rather than assuming every average gives the same kind of evidence.
- When comparisons can shift
For model comparisons, the framework measures the distance between a test’s difficulty distribution and a chosen target. Its theory says that when the performance difference between two models changes smoothly across difficulty, this distance bounds how much the test can distort their score difference.
If the target difference exceeds that bound, the ranking is preserved. The claim is conditional: it does not guarantee agreement for smaller differences or when the smoothness assumption fails.
- Where improvement goes unseen
A separate diagnostic asks whether questions cover the difficulty range where improvement occurs. The authors define the largest interval with no question and show that an improvement confined to an interval no wider than that gap can be undetectable in the worst case, even with repeated evaluations.
In their analysis of 25 tests, eight had a largest gap of at least 0.10 on the normalized difficulty scale.
- Diagnosing and filling gaps
The authors estimate difficulty from model responses and apply both diagnostics across tests in seven domains. Ten of the 25 were at least 0.10 away from a same-size uniform difficulty reference, though unevenness is not inherently a problem if it suits the evaluation goal.
They also test BenchFill, which adds questions aimed at gaps: across eight tests, it reduced the three targeted gaps more than a rewriting baseline with the same proposal budget.
Deep
- Original paper authors
- Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang
- Submitted to arXiv
- License
- CC BY 4.0