Deleting agent history works best when recent tool results stay
The Hidden Costs of Forgetting: A Study of Trajectory Deletion in Software Engineering Agents · arXiv:2610.35496
In plain words
Software engineering agents solve coding tasks by taking turns: they reason, use tools to inspect or change files, and receive results. This paper studies what happens when some of that history is deleted before later turns.
The researchers tested two agents on 100 coding tasks, changing what they deleted and when. Keeping the latest two turns intact often avoided a statistically significant drop in tasks solved, but runs could take longer. The results apply to the tested agents, model and tasks.
The research
When a coding agent’s history grows, removing old information may seem like a way to reduce what it has to process. But does deletion preserve task solving, or does the agent spend extra turns recovering what it lost? The answer matters for judging both effectiveness and estimated execution cost.
The study varies what an agent forgets and when, showing that delayed deletion often preserves task solving while costs and run length can rise.
They varied what history was removed, how much, and whether deletion was immediate or delayed by two turns. Across two agents and 100 tasks, 37 of 42 delayed-deletion evaluations showed no statistically significant drop in tasks solved, though 21 had significantly longer runs. All 21 deleted tool output.
The comparison suggests that a delay can protect task solving, while extra work may remain. When tool output was deleted, agents often retrieved it again, and estimated savings depended on how repeated prompt text was billed.
These results come from one model, two agents and 100 coding tasks. Each deletion configuration had one run per agent on each task, with one random deletion draw for each run. Passing tests is a proxy for patch correctness, and the cost estimates depend on simplified cache assumptions.
What the evidence shows
- The question and test
The study asks how removing parts of an agent’s accumulated work history changes task solving, later behavior and estimated execution cost. It tests two software engineering agents on 100 SWE-bench Verified tasks, coding tasks based on real GitHub issues where a solution counts as successful if its patch passes hidden tests. The authors also provide a dataset and replication package containing 9,400 trajectories.
- What deletion changes
The researchers varied which content could be deleted, how much was removed, whether deletion affected whole turns or randomly chosen tokens, and whether it happened immediately or after two turns. For each configuration, they compared results with uncompressed runs from the same agent on the same tasks. This design lets them examine timing alongside the other deletion choices, within the tested setup.
- Delay protects task solving
Keeping the latest two turns intact was associated with fewer task-solving losses than deleting content immediately: 37 of 42 delayed evaluations showed no statistically significant decrease in tasks solved, compared with significant decreases in 19 of 42 immediate evaluations.
These counts are across both agents. Delayed deletion still significantly lengthened runs in 21 evaluations, all of which deleted tool output; immediate deletion lengthened runs in 34.
- Deleted results are retrieved
The researchers examined file reads to see whether agents recovered information removed from their history. In 24 evaluations of turn-level deletion configurations across the two agents and deletion timings, deleted read outputs were re-fetched 1.5 to five times as often as retained outputs.
The difference was significant in 23 comparisons. This supports the interpretation that some agents retrieved deleted information again, which can help explain longer runs.
- Cost depends on pricing
Fewer prompt tokens did not guarantee lower estimated costs. Under the study’s caching-aware pricing assumptions, none of the 42 delayed-deletion evaluations cost less than the uncompressed reference, and 26 were significantly more expensive.
Rewriting history can break the discounted matching prompt prefix; extra turns can add further costs. Under flat pricing, 26 estimates were lower, but only six reductions were significant.
Deep
- Original paper authors
- Fernando Vallecillos-Ruiz, Islem Bouzenia, Michael Pradel, Leon Moonen
- Submitted to arXiv
- License
- CC BY 4.0