What Tests and Bug Checks Tell Us About Software Agents

An agent may deliver a package that passes many tests or a patch that stops a known crash. Those are useful results, but the checks may see only a supplied interface, a particular input, or a known defect.

Across four studies, execution helps agents revise code and expose failures, while the test setup determines what success records. The supported answer is bounded: each result speaks to behavior observed in its own task and evaluation.

Suppose an agent must rebuild a Python package from its public interface. The files have to import together, agree on shared data, and pass the package's tests.

A test run gives the agent a concrete failure to investigate, and a final score can show how much tested behavior the reconstruction recovered. Yet the package may have behavior the supplied tests never exercise. Before trusting its delivery, a reader needs to know which tests ran and what the score compares.

Now consider an agent patching a vulnerability. The known input that exposes the bug may stop crashing after the edit, while the edit also changes behavior the project's developers intended to keep.

A benchmark can run that known input, the existing regression suite, and tests added with the upstream fix; these checks answer different questions about the patch. A reported pass rate makes sense only alongside the check that produced it and the conditions in which the agent worked.

A concurrency bug presents a different difficulty: a crash may appear only when two threads execute particular operations in a rare order. A code reviewer faces yet another one: the defect may depend on project knowledge beyond the changed lines.

These cases sharpen the shared question. When software agents use tests, controlled execution, or records of known bugs, what have they shown about a change, and what remains outside the evidence?