Inherited software work exposes a gap in coding agents
SWE-Residual: Evaluating Starting-State Robustness of Coding Agents · arXiv:2610.25899
In plain words
Coding agents read and edit code or operate a computer to complete a task. Most tests give them a clean starting point, so they do not show whether an agent can finish work that another agent left incomplete.
SWE-Residual is a test set where agents continue from earlier work that was submitted as done but still failed hidden tests. Across five tested agents, success was lower from these inherited states than from matched clean starts on both repository and machine tasks.
The research
When a coding agent inherits work that another agent marked complete, can it identify what is still wrong or missing? Clean-start tests do not answer that question, because they give the agent a task without earlier work on it.
SWE-Residual is a benchmark that tests whether coding agents can finish tasks from earlier work that was submitted as complete but remains faulty.
The authors created SWE-Residual, a benchmark of earlier attempts that were submitted as complete but still failed the original hidden tests. Evaluated agents received the original request and inherited state without an explicit failure verdict; matched Clean Starts used the same task conditions.
All five agents had lower success from inherited states on both repository tasks and machine-environment tasks. Since the request, verifier, harness, prompt, and budget were matched, the results show that success on clean starts did not carry over reliably to these inherited states.
The measurements cover 198 states from 111 tasks, drawn from two source tests and three predecessor model families. The acceptance-bias analysis is limited to DeepSWE, where the needed per-test outcomes and final code changes were available; other kinds of handoffs were not tested.
What the evidence shows
- A missing handoff test
Most coding-agent tests begin from a clean repository or machine. That measures whether an agent can do a task from scratch, but not whether it can recognize what remains incomplete in earlier work submitted as done.
SWE-Residual is a benchmark for this separate ability: maintaining task performance when the starting state already contains work on the same request.
- Making handoffs testable
The authors built SWE-Residual from predecessor attempts that were submitted as complete but still failed the original hidden-test verifier. They reviewed, reconstructed, and validated the resulting states.
The collection contains 198 such states from 111 tasks across repository tasks and machine-environment tasks, with predecessor work from three model families.
- A matched starting comparison
In a Residual Start, the evaluated agent receives the original request and the inherited state, but no explicit trajectory or failure verdict. The matched Clean Start uses the same request, verifier, harness, prompt, and budget.
Since the starting state is what changes, the comparison tests whether earlier work affects success under this protocol.
- Success dropped in both settings
All five evaluated agents had lower success from Residual Starts than from matched Clean Starts. The gap ranged from 20.6 to 53.5 percentage points on DeepSWE repository tasks and from 20.6 to 34.9 points on Terminal-Bench machine tasks.
The authors averaged outcomes within each task, then weighted source tasks equally, so these ranges summarize agents’ results rather than every task’s outcome.
- Inherited failures persist
On DeepSWE, many failed continuations ended with exactly the same hidden tests failing as at the start. Yet agents passed many of those same initially failing tests on the same tasks from Clean Starts.
The authors call the recurring pattern acceptance bias: inherited work often shapes what agents treat as remaining, while omitted files stay unchanged. This analysis helps explain the performance gap, but covers DeepSWE only.
Deep
- Original paper authors
- Jinze Li, Weijie Liu, Jinfeng Xu, Shuo Yang, Xin Yang
- Submitted to arXiv
- License
- CC BY 4.0