In plain words
Coding agents may change an existing software project without meeting every part of the request. PACE-Bench is a way to check whether they delivered the whole request, including business requirements and connections to existing code.
It turns real development requests into expert-reviewed checklists, then checks agents’ code against them. In its 43-task comparison, the best configuration fully met the acceptance rule in 6 cases, so complete delivery was uncommon in this test set.
The research
A coding agent can pass tests for individual components while leaving a business rule or a connection to existing code unfinished. For requests that specify several acceptance conditions, how can an evaluation tell whether the whole request was delivered and identify what remains unmet?
PACE-Bench checks coding agents’ changes against frozen, expert-reviewed requirements and links unmet conditions to evidence from the code and execution traces.
In mature software projects, a coding agent can pass isolated checks yet leave a business rule or integration requirement unmet. How can an evaluation judge the whole request and show which conditions remain missing, rather than compressing different gaps into one score?
PACE-Bench checks patches against frozen, expert-reviewed acceptance criteria and links unmet requirements to code and execution evidence. Across 43 tasks, the top reported configuration met the complete-delivery rule in 6 cases.
A separate evaluator check matched technical-annotator consensus on 95.2% of 750 verdicts for Qwen Run 1 patches.
These results describe the 43 production tasks and the tested configurations. The release comparison reports one Opus-4.5 configuration per release, so it does not show that harness changes alone caused the score increase; the task set also does not cover alternative sources such as merge requests or repository structure.
Analysis
Deep
- Original paper authors
- Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang, Gang Yu
- Submitted to arXiv
- License
- CC BY 4.0