An escalation option helps coding agents stop at repair barriers
Knowing When to Stop: The Evidence-Action Gap in Coding Agents · arXiv:2610.31895
In plain words
Coding agents can run into problems they cannot fix themselves, such as a repair environment refusing changes. This paper introduces Impasse Bench, a set of software repair tasks with controlled blockers, to measure whether agents act sensibly as they gather clues.
In the tested persistent-barrier tasks, most runs collected enough evidence, but many still used all 80 steps. Adding a way to ask for help reduced these endings across the tested runs, though some agents also asked for help when a temporary fault would have cleared on retry.
The research
When a coding agent’s tools show that an environment is blocking a repair, how can we tell whether it has enough evidence to stop, and whether it will act on that evidence? A useful measure must distinguish a premature handoff from continued attempts after a persistent barrier has become clear.
Impasse Bench scores coding agents against the evidence they gather, and paired tests show an escalation option can curb over-persistence on persistent barriers.
The authors paired runs with and without an escalation tool and instructions for handing off persistent infrastructure barriers. In 4,271 pairs with sufficient evidence in both settings, mean valid tool calls after that point fell from 38.9 to 17.5.
In the full cohort of 4,344 persistent-barrier pairs, endings at the 80-step limit or after three consecutive text-only replies fell from 33.5% to 6.6%. This supports the conclusion that a handoff option can help agents stop after recognizing these barriers, though some models rarely used it.
These results come from controlled faults in sandboxed repair tasks and specified evaluators. They do not establish safety in deployment or outside the behaviors and environments measured.
What the evidence shows
- The stopping problem
Impasse Bench is a set of software repair tasks for measuring whether coding agents act in line with their tool feedback when environment faults block progress. The tasks remain solvable, but a controlled fault can prevent a repair, limit verification, or clear on retry.
The authors call a mismatch between evidence and action the evidence–action gap: an agent may keep trying after clues show it needs outside help, or stop before reasonable recovery attempts.
- Anchor actions to evidence
To separate those two mistakes, the authors pair each fault with an evaluator-only card that defines expected signals, checks, recovery attempts, and an appropriate response. A rule-based annotator uses the card and recorded tool feedback to mark the first round when sufficient evidence has been collected.
The point anchors judgments about what happens next; it does not assert that repair is impossible or every recovery has been exhausted.
- Evidence often came first
With the basic interface, 98.6% of 4,344 persistent-barrier runs collected sufficient evidence, yet 33.5% reached the 80-step limit. Every budget-exhausted run in this cohort had already collected enough evidence.
This indicates that, in these tested cases, many agents did not stop after recognizing the barrier. The result is an aggregate across eight model configurations, not a description of each model or run.
- A handoff changes behavior
The authors compared paired runs with and without an escalation tool and instructions to hand off persistent barriers. In the full persistent-barrier cohort, endings at the 80-step limit or after three text-only replies fell from 33.5% to 6.6%.
Among pairs with enough evidence in both settings, mean valid tool calls after that point fell from 38.9 to 17.5. The results support the view that an explicit handoff option can help agents act on evidence in these tasks.
- Stopping well takes judgment
The results also show why stopping sooner is not enough. Ten tool-enabled runs handed off during a temporary sandbox fault that recovered after two failed calls; all had empty patches and failed target tests.
A separate harness comparison found higher repair success after scripted restoration when evidence-based reminders and a ticket template were added together. Ticket ratings further showed that a handoff can name the blocker while leaving the agent’s progress unclear.
Deep
- Original paper authors
- Tianyu Chen, Xinran Zhao, Vijay Viswanathan, Shoubin Yu, Mingyuan Zhou
- Submitted to arXiv
- License
- CC BY 4.0