Why vulnerability patch scores can miss the intended fix

When a software patch passes an automated test, does that mean it fixes the vulnerability in a way that preserves intended behavior? A check that asks only whether a known input still crashes can be passed by suppressing that symptom.

This paper asks how evaluation choices shape reported patch scores and whether a broader check changes what those scores mean.

The authors compare patch-evaluation setups and add upstream developer tests to reveal when crash-test scores miss intended behavior.

The authors compare patch-evaluation setups to ask whether reported pass rates reflect more than stopping a known crash. They add tests from upstream developer fixes to check behavior around the bug, alongside build, crash, and existing regression checks.

Across 112 cases, pass rates changed with prompt information and test format. In a model comparison, Opus 4.7 passed the crash check on 90.2% of cases, but 32.7% of those crash-passing cases failed the developer test. The two checks therefore support different conclusions about a patch.

The dataset contains 112 carefully selected bugs from three language families, and developer-added tests are only an imperfect proxy for what maintainers would accept. Passing those tests is evidence of closer alignment with maintainer intent, not proof that a patch is correct.