Match a test verdict to the failure it actually checks

A passing generated test may look reassuring even when its expected result comes from faulty code. A test checked against one repaired program may also misjudge other valid implementations, while a thread failure may stay hidden until a particular execution order occurs.

Across three studies, the useful question is whether the check was placed in a position to distinguish the failure at issue. Their evidence comes from injected code faults, curated implementation panels, and controlled thread executions; each verdict stays tied to those evaluated conditions.

A pull request changes a calculation, and an automated tool adds a test for the edited lines. The test runs and passes.

A developer might read that green result as evidence that the new behavior is right. Yet the regression-test study examines changes that already contain faults missed by the existing test suite.

In that setting, a passing generated test shows agreement with the changed code. The developer still needs evidence about the behavior the change was meant to provide.

An assertion drawn from the current implementation can turn a bug into an expected result. The regression-test study calls a test fault-enforcing when it passes on faulty code and fails on the corresponding version without that fault.

Such a test may enter a project quietly, then make a later correction look like a regression. Its immediate green result conceals the practical question: did the check challenge the relevant failure, or repeat the behavior it observed?

The same question appears in other forms. A suite might pass on one repaired program yet reject another implementation that meets the requirement.

A fault involving two threads might remain unseen because the threads never ran in the order that exposes it. These cases involve different tests and different failures, but each verdict comes from particular programs or executions.

To interpret the verdict, a reader needs to know which failure the check had an opportunity to distinguish.