Generated tests can preserve bugs as expected behavior
In plain words
A regression test is meant to catch a bug if it returns. But a test generator that sees buggy code may treat the bug as normal: its test passes while the bug is present and fails after a fix.
The researchers injected artificial faults into code from 145 pull requests in three Python projects, generated tests, and checked them against faulty and fixed versions.
Fault-enforcing tests were more common than tests that caught the injected fault in all three projects, and most tracked fault-enforcing tests kept accepting their fault through the studied history.
The research
A generated test can pass on buggy code and fail when the bug is fixed, making the bug look like intended behavior. Do these tests disappear as a project changes, or can they keep that expectation in place? The answer matters for interpreting what passing generated tests establishes.
The authors inject artificial faults into historical changes, classify generated tests by their behavior on faulty and fixed code, and track fault-accepting tests over time.
The authors injected faults into historical pull requests, generated tests without showing the model the fixed code, and compared each test’s outcomes on faulty and fixed versions. In SciPy, Qiskit and pandas, tests that accepted the fault were more common than tests that exposed it.
They then followed fault-accepting tests through later changes. Most still enforced their fault at the end of the selected histories, supporting the concern that generated tests can preserve faulty expectations as code evolves.
The study uses artificial mutations in three Python projects and one test generator, so its rates do not establish how often real faults or other systems would behave this way.
Analysis
Deep
- License
- CC BY 4.0