BestRand selects smaller tests to steady model scores
Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand · arXiv:2611.00883
In plain words
Checking every item in a large test can take substantial time and computing power. This paper introduces BestRand, a way to estimate a model’s full-test score from fewer items: a few probe models help choose a subset that looks representative, and the target models are scored on that subset.
Across 70 text, multimodal and agent tests, BestRand reduced average score-estimation error compared with three other subset methods. The result is an average across tests and item budgets, so it does not mean every subset gives an accurate estimate.
The research
A compressed test is useful only if its score still tracks the full set; occasional bad samples can make estimates unreliable. Understanding how subset choice affects that error helps show how much confidence a smaller evaluation can support.
BestRand uses a few probe models to choose a candidate subset whose scores resemble the item pool, then evaluates target models on that smaller set.
A few probe models score a pool of items, and BestRand picks a candidate subset whose average probe scores are closest to the pool average. Target models are scored only on that subset, and their estimates are compared with scores from the full benchmarks.
Across 70 benchmarks and six item-retention levels, BestRand had lower average absolute score error than three subset methods. This supports probe-guided selection as a way to improve estimates on average.
The theory’s account of probe matching depends on an idealized selection model, and BestRand can still make estimation errors. The tests cover 70 text, multimodal and agent benchmarks; a separate extension covers two vision-language-action benchmarks.
What the evidence shows
- The smaller-test question
BestRand is a benchmark-compression method: it selects a smaller set of items to estimate how models would score on a full benchmark.
The question is whether those estimates remain accurate and reliable when evaluation is reduced, since checking every item can require substantial time and computing power.
- A shared source of error
The theory treats a benchmark as items scored by multiple models. It identifies a shared shift when a sampled subset is easier or harder than the full benchmark, moving many models’ estimates together.
Under its assumptions, this shared shift helps explain why most random subsets can have small errors while a few produce much larger ones.
- Probe scores guide selection
BestRand uses probe models to score a pool, then chooses from candidate subsets the one whose average probe scores most closely match the pool average. Target models are evaluated only on the chosen subset.
The theory suggests this can reduce the part of the shared shift visible to probes, but that explanation relies on an idealized selection model.
- Lower average score error
Across 70 text, multimodal and agent benchmarks and six item-retention levels, BestRand reduced average mean absolute error by 16.4% versus Random, ZipBench and TinyBenchmarks. Mean absolute error is the average absolute difference between a subset-based score estimate and the full-benchmark score.
The comparison averages across benchmarks and retention levels, rather than showing an improvement on every one.
- Fewer high errors and transfer
The comparisons examine two limits of compressed scores: occasional unusually large errors and accuracy when construction and test models differ. Across the tested benchmark categories and settings, BestRand had the lowest risk of errors exceeding thresholds set by Random, and at 20% retention it had the lowest average error in each of five score-range, model-family, and time-shift settings. These aggregate findings do not guarantee an improvement on every benchmark or shift.
Deep
- Original paper authors
- George Ling, Shanshan Zhong, Lijin Yang, Hao Yang, Nanxiang Li
- Submitted to arXiv
- License
- arXiv non-exclusive