Coding agents often lose feature correctness across stages
FeatureArc: Benchmarking Coding Agents on Evolving Features · arXiv:2610.37434
In plain words
FeatureArc is a task set and construction pipeline for testing whether coding agents can keep a software feature working as they add requirements to their own earlier code. It draws stages from repository histories and checks both the new behavior and behavior that should keep working. Agents get no feedback about failed stages.
Across ten configured agents, 62.7% of tasks passed the first-stage check, while 28.5% passed the final check of the complete feature. This pooled result shows that passing an initial request often did not mean the full evolving feature worked.
The research
A single coding task shows whether an agent can meet one request, but not whether later changes preserve earlier behavior. When agents build on their own code without being told a stage failed, does early success carry through to the complete feature?
FeatureArc turns repository histories into sequential coding tasks with hidden checks, then measures whether agents keep the whole feature working.
FeatureArc derives sequential requirements from repository histories and checks each stage for the requested new behavior and preserved existing behavior.
At the end, a separate check tests the complete feature on the agent’s final code; ten configured agents worked in one persistent session without failure feedback or rollback.
Pooling tasks across agents, 62.7% passed the first-stage check, compared with 28.5% passing the final whole-feature check. The gap shows initial-stage success often failed to carry through.
In a separate within-task-and-stage comparison, next-stage success was also lower after a failed prior stage, while trajectory labels linked many later failures to inherited defects.
The comparison after a failed stage is conditional and does not show that the earlier failure alone caused later outcomes. The 73 tasks were a manually reviewed subset from 53 repositories, and the authors say FeatureArc may not cover the full range of real-world feature-development workflows.
What the evidence shows
- The risk of extending code
A coding agent may handle one request correctly while leaving a defect in its code. If later requirements build on that code without the agent being told a stage failed, the defect can affect the rest of the feature.
FeatureArc is a benchmark and task-construction pipeline designed to test whether agents can keep a feature working as they extend it through successive requirements.
- Tasks follow feature history
FeatureArc recovers successive changes to one feature from repository histories and turns them into sequential tasks. Each stage specifies expected behavior and an interface.
The task checks new behavior and behavior that should remain intact, then uses a separate final check for the complete feature. In the evaluation, agents stayed in one session and workspace and received no feedback about failed stages or rollback.
- Initial success often does not carry
The first-stage check measures the initial request, while the chain gate checks the complete feature on the agent’s final code. Across ten configured agents, 62.7% of tasks passed the first-stage check and 28.5% passed the chain gate.
Because these are pooled rates over the same task set, they show a gap between initial and whole-feature success without implying that every agent or task followed the average.
- Earlier defects affect later work
A within-task-and-stage comparison found a lower mean next-stage pass rate after a failed stage than after a passed one: 29.0% versus 72.6% across 109 groups containing both outcomes. Separately, trajectory labels identified inherited defects in 72.9% of failed later stages.
These findings link earlier failures to later breakdowns, though the comparison does not establish that prior failure alone caused them.
- Self-checks can miss errors
The study also examined whether agents recognized earlier failures and whether their own checks signaled correctness. Trajectory labels indicated that agents never noticed the earlier error in 94.9% of stages following a failure.
Among 713 such later stages that ended with a passing self-test, only 24.5% passed the hidden test for new behavior. These results point to limited error detection and self-checking in this evaluation.
Deep
- Original paper authors
- Yunlong Wang, Alyssa Wu, Cuiyun Gao, Jia Feng, Songyuanyi Lu
- Submitted to arXiv
- License
- arXiv non-exclusive