Agent benchmarks can stop ranking before tasks run out
Agent Benchmarks Stop Ranking Before They Run Out of Tasks · arXiv:2611.13541
In plain words
Data-science agents are often ranked by whether they pass a task’s tests. Many can receive the same pass mark, and a score that stops at a reference cannot reward work that goes beyond it.
The paper proposes an evaluation method that keeps the pass check, quality score, and cost separate. On ALE-Bench programming contests, graded scoring distinguished more groups among the top agents than a median-human pass/fail score did on the same released runs. This finding is limited to those runs.
The research
A leaderboard can stop separating strong systems even while their task results still differ. That changes what a flat ranking means: it may call for better measurement on the same tasks, rather than automatically harder tasks. Pass rates then show threshold crossing, not necessarily fine-grained ordering.
The paper proposes an evaluation methodology that separates valid completion, graded task quality above a reference, and reported cost for data-science and machine-learning agents.
The authors rescored released runs and compared scoring rules on the same tasks.
On ALE-Bench, which tests agents on programming contests, the official contest score and a graded score each split the top ten into three tiers; a median-human pass/fail score and a score capped at that reference each produced one tier.
This comparison shows that a single threshold or a cap can hide distinctions in these runs, while graded credit can retain them when agents exceed the reference.
The ALE-Bench tier comparison uses the same released runs on 40 contests. It shows finer separation on that benchmark, not that graded scoring resolves every benchmark.
What the evidence shows
- A ranking can saturate early
The proposed agent-evaluation methodology separates a pass check, a quality score that can reward results above a reference, and a cost measure. It addresses two limits of one-bit pass/fail rankings: nearby systems can receive the same verdict, and systems that exceed the reference can still tie. A flat leaderboard may therefore reflect the scoring rule as well as task difficulty.
- Stronger tests change a tier
SWE-bench Verified tests agents’ software patches, and its score records which patches pass. Among its official top 30 agents, rechecking passing patches with SWE-ABS strengthened tests reduced the top tier from 23 agents to five.
The agents leaving that tier remained nearly tied with one another, so the sharper check changed tier membership without resolving neighboring ranks.
- Graded credit keeps distinctions
ALE-Bench tests agents on programming contests; its tier count measures distinguishable groups among the top ten, with more tiers meaning finer ranking. On the same released runs across 40 contests, the official contest score and a graded score each produced three tiers.
A median-human pass/fail mark and a score capped at that reference each produced one. The comparison isolates how scoring changes ranking on this task set.
- Higher bars shift the band
The authors also compared pass/fail bars on ALE-Bench models split by release date. Among the later cohort’s top ten, the median-human, earlier-tuned, and calibrated per-task bars each produced one tier; the contest-winner bar produced two, while 89% of earlier models scored zero on every task.
A graded score with six bars per task produced three tiers among the later top ten. Raising a single bar can shift which group it distinguishes.
- Pass marks can hide quantity
On Terminal-Bench 2.0’s portfolio-optimization task, tests tied Claude Opus 5, GPT-6 Astra, and Gemini 3.1 Pro because all their runs passed. Across three trials per model, median speedup relative to the reference solution was 10.9x for Claude Opus 5 and 1.3x for the other two, matching the reference’s level. The comparison shows how a pass verdict can hide differences in a measured task result.
Deep
- Original paper authors
- Zihan Dong, Qishi Zhan, Minxuan Hu
- Submitted to arXiv
- License
- CC BY 4.0