In plain words
When a coding agent gets a program wrong, it might need more time to think, useful information about what the program should do, or feedback from running tests. It is hard to tell which helped if the agent’s work changes at the same time.
The paper introduces a way to compare these resources from copies of the same failed attempt. Across function-level coding tasks, descriptions of the required behavior usually helped more than test feedback for the six repair models tested. The study measured one revision at a time.
The research
When a coding agent is stuck, which resource actually helps its next repair: more computation, information about what the code should do, or feedback from tests? In a changing repair trajectory, these inputs are difficult to separate from changes in the agent’s state and revision.
BRA clones the same failed coding-agent state across resource combinations, then compares each resource’s average contribution to one final revision.
BRA compares resources by giving copies of the same failed coding-agent state different combinations of computation, information, and feedback. The same model makes one final revision for each combination, and the study scores revisions on held-out tests. This design lets the authors estimate each resource’s contribution at a fixed state.
On function-level failures, audited specifications of required behavior usually contributed more than public-test feedback. On repository failures, retrieved code contributed little, while a reference-derived specification helped more on the same states.
The results point to the content of information as important, but apply to the study’s frozen failures and one-revision setup.
These results measure one revision from frozen failures. The repository specification was derived from reference fixes, and the study does not establish how feedback works across a multi-step repair trajectory or what information an agent can obtain in practice.
Analysis
Deep
- Original paper authors
- Saiyue Lyu, Jerrold Huang, Yunbei Zhang, Janet Wang, Zhenting Wang
- Submitted to arXiv
- License
- CC BY 4.0