A bug-based test finds limits in automated code review
In plain words
CRBugBench is a test of whether code review agents can find bugs that people later confirmed and fixed. Instead of checking whether an agent’s comments resemble human comments, it gives the agent the original code change and repository context, without saying a bug is there.
On GitHub tasks involving bugs people caught before the change was merged, claude-opus-5 included the known bug in its first five findings on 79.6% of tasks. The test measures bug detection, not every part of a good review.
The research
If human review comments can miss valid bugs, how can developers tell whether automated reviewers recover defects that people actually confirmed and fixed? A useful test needs to check the defect itself while giving agents the repository and change they would have reviewed.
CRBugBench pairs confirmed bugs with the changes that introduced them, then measures whether review agents recover those bugs in repository tasks.
For each task, an agent reviews the repository snapshot and pull request that introduced a known bug, without being told a bug is present. Three language-model judges label whether its findings match the bug, and the researchers calculate the share of tasks where it appears in the first five.
On GitHub tasks, claude-opus-5 recovered bugs caught before merge more often than bugs found after merge: 79.6% versus 64.2% of tasks. The results show strong but incomplete recovery, with performance differing by track.
These results measure recovery of known bugs on selected repository tasks, not overall review usefulness or performance across all software projects. The adjusted associations describe patterns in the data; they do not show that bug visibility alone causes detection differences.
Analysis
Deep
- License
- CC BY 4.0