Generated tests often fail across valid implementations
En palabras sencillas
The paper asks whether coding agents write tests that separate correct programs from incorrect ones. A common check uses the original program and one reference answer, even when several implementations can be valid.
TestPrism adds valid and invalid alternatives. TestHelix creates several test-and-repair pairs, checks them against one another, and improves its generation strategy.
Its final workflow beat native agents on the paper’s 300-task test set for both model groups, but succeeded on only about one-third of tasks.
La investigación
When a test passes one reference implementation, does that show the test captured the task’s required behavior, or only that it fits that particular program? Generated tests may reject valid alternatives or accept invalid ones, making a system appear more reliable than its tests actually are.
TestPrism checks generated tests against valid and invalid alternatives, while TestHelix adds peer checking and strategy refinement to improve reliability across implementations.
TestPrism pairs 300 test tasks with one reference, four other valid implementations, and five invalid ones. Each submitted test is rerun on the starting program and every candidate; its task score requires failing first, accepting all valid candidates, and rejecting all invalid ones.
For Claude Fable 5.1, the full-panel score was 28.00%, versus 59.67% for reference-only checking. With tests fixed, adding candidates also lowered the mean score, supporting the authors’ interpretation that one-reference success can overstate reliability across implementations.
The evidence is limited to fixed, balanced candidate panels and validated task contracts in TestPrism. It does not establish exhaustive semantic coverage, real-world patch prevalence, or performance on evolving specifications and broader execution environments.
Análisis
Detalles
- Autores
- Han Li, Lingxiang Hu, Jiacheng Huang, Ziqian Jiang, Jingkai Luo
- Licencia
- CC BY 4.0