A code-history check flags overlap in bug-fixing tests

Bug-fixing tests use real software defects to compare how often systems make successful repairs. But if a model’s training material already contained the repository or fix, does its score show how well it handles the defect, or partly reflect familiar code? This is especially hard to check when only written model answers are available.

BenchChecker combines textual code probes with repository history to flag bug-fixing samples whose fixes may have appeared in a model’s training data.

The authors tested BenchChecker on three models trained on The Stack v2, where sample membership was known, across six bug-fixing tests. Four comparison methods relied on internal model scores; BenchChecker used written answers and repository history.

Its AUC values were around 0.9, measuring how well its scores separated included from clean samples, and its false-positive rate stayed at or below 0.1, so no more than one in ten clean samples was flagged.

On leaderboard submissions, resolution rates fell after samples identified as contaminated were excluded: more than 20% for every top-ten SWE-bench Verified submission on the 1–4-hour subset, and about 40% on average for the top ten on Multi-SWE-bench’s Medium subset.

The authors read these drops as evidence that original results overstate performance on those tasks.

The second check depends on the fix still being visible in the repository snapshot from before training ended. BenchChecker can miss fixes removed or substantially changed by then, and it does not directly detect overlap through issue descriptions or discussions.