A continuing audit finds defects in released agent benchmarks
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks · arXiv:2611.08417
In plain words
Released tests for software agents can become unreliable when task instructions, grading checks, or outside dependencies change.
This paper introduces a workflow that gathers reports and past agent attempts, asks software auditors to identify problems and suggest repairs, then checks those suggestions before maintainers decide what to accept.
On Terminal-Bench 2.0, the audit flagged 28 of 89 tasks for repair. After the update, the overall ordering of 19 agents stayed similar, although some scores and ranks shifted.
The research
Scores from released agent tests can shape comparisons between systems, yet task defects may make those scores diverge from the abilities the tests are meant to measure.
This audit shows that repairs can leave broad agent rankings similar while shifting individual scores, so conclusions about close comparisons may depend on whether tasks have been checked and updated.
A continuous-validation workflow gathers and checks evidence of task defects, proposes repairs for human review, and updates Terminal-Bench while tracking how scores change.
The workflow audited Terminal-Bench 2.0, then researchers compared 19 agents on the old and repaired versions under the default protocol. The rankings remained strongly correlated, but individual scores and leaderboard positions shifted.
This indicates that repairs can preserve broad ordering while changing absolute scores and close comparisons. A separate Harbor-Index sample also provided transfer evidence, alongside examples of defects the workflow missed.
The agent comparison covers 19 agents under the default protocol, and the Harbor transfer check covered only 34 randomly selected tasks from one 80-task snapshot. Harbor reviewers also found defects the workflow had missed.
What the evidence shows
- Keep task scores meaningful
The continuous-validation workflow is a method for finding and checking defects in released agentic benchmarks, which test whether software agents can complete tasks using computer tools.
Its purpose is to keep task scores tied to the abilities the tasks are intended to measure as instructions, environments, and dependencies change.
- A substantial repair backlog
The audit shows that post-release review can uncover repair needs after a benchmark has launched. Because the total combines earlier reports with newly identified tasks, it describes the maintenance backlog rather than the workflow’s independent discovery rate.
- Similar ordering, shifting scores
The update preserved broad ordering across the evaluated agents, but some individual scores and nearby leaderboard comparisons changed. A stable overall ranking therefore does not guarantee that specific score gaps or positions are unchanged.
- Time changes a harness comparison
For GPT 5.4 on Terminal-Bench 2.1, most of the default-time score gap between Codex and Terminus-2 narrowed when the time limit was extended.
This additional test covered only a subset of agents and departed from benchmark guidelines, so it shows that this comparison depends on time allowed, not how harnesses generally compare.
- A sample beyond Terminal-Bench
Manual review of a small random Harbor-Index task sample confirmed that many of the workflow’s flags were real defects, providing evidence beyond Terminal-Bench.
Reviewers also found defects among tasks it had not flagged, so this transfer check supports its use for triage but does not show that it catches every defect.
Deep
- Original paper authors
- E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Ré
- Submitted to arXiv
- License
- CC BY 4.0