Vulnerability fixes must also keep projects working

When a dependency has a known security fix, can a coding agent apply it without leaving the vulnerability in place or breaking the project? Upgrade checks that look only at whether builds and tests still work can miss an incomplete fix. The paper asks how to evaluate both sides of the task together.

VulnBump tests real dependency upgrades with an independent four-check verdict, showing which coding systems remove vulnerabilities while keeping projects working.

The authors introduce VulnBump, which tests 307 dependency upgrades in 43 repositories across six package managers. An independent verifier checks in a fresh container that dependencies resolve, a rescan finds the vulnerability fixed, the project builds, and previously passing tests still pass.

Using the same model, Snyk’s Evo Remediation Agent passed all four checks on 90.2% of upgrades, versus 77.5% for Claude Code with the same Snyk findings and target. The joint score measures remediation and project integrity together.

The rates are pooled across upgrades, not repository averages. Evo Remediation Agent and Claude Code use different harnesses, prompts, and tools, so the comparison does not identify which component explains their difference; each system ran once and was checked with each project’s own tests.