AT-AT improves shared tools from agents’ past runs
AT-AT: Automatic Tool Optimization from Agent Traces · arXiv:2610.34679
In plain words
Agents use tools to get information and take actions, but the people who maintain shared tools may not control the agents that use them. AT-AT is a method that reads evaluated past runs and uses a coding agent to edit tool instructions and, when allowed, the code behind them.
On held-out simulated customer-service tasks in Telecom, the test’s default agent had a higher mean task-completion reward with AT-AT’s tools than with the original tools. The result is an average across tasks and repeated runs, not a guarantee for each task.
The research
When many agents depend on shared tools, can a provider improve their results using past runs without changing the agents? The question matters because descriptions guide which tools and arguments agents choose, while tool implementations determine the information and errors returned after a call.
AT-AT uses evaluated agent runs to guide selective edits to shared tool descriptions and code, aiming to improve outcomes while agents stay fixed.
The authors ask whether shared tools can be improved using agents’ past experience when tool providers cannot change the agents themselves. Tool descriptions affect what agents call, while tool implementations shape the information and errors they receive.
AT-AT gives a coding agent evaluated traces and tool source so it can find recurring failures and selectively edit descriptions and, when allowed, implementations. On held-out simulated customer-service tasks in Telecom, the evaluator’s mean task-completion reward rose from 29.50% with original tools to 36.75% with AT-AT tools. The paired comparison across tasks and repeated trials supports a gain in this setting.
The benchmark result comes from one optimized toolset per optimizer, so it does not show how results vary across optimization runs. The production evaluation covered three selected servers during one period, and repeated evaluations across deployments and time are needed to establish robustness.
What the evidence shows
- Improve tools, keep agents fixed
AT-AT (Automatic Tool Optimization from Agent Traces) is a method for improving shared tools from agents’ evaluated past runs while keeping the agents fixed. This addresses a practical constraint described by the authors: tool providers may maintain tools used by many agents without being able to change those agents.
A tool’s description shapes what an agent calls and how; its implementation shapes the information and errors returned after a call.
- Use past runs to guide edits
The method gives a coding agent evaluated traces and tool source, then has it search for recurring failure patterns and make selective edits. It can change descriptions and, when allowed, implementations.
This distinction matters because better instructions can shape a call before it happens, while changed implementation code can shape the information returned afterward. The edits are made offline from existing runs, without new agent executions during optimization.
- A Telecom gain on held-out tasks
The clearest measured gain came from τ²-bench-verified, a test of agents completing simulated customer-service tasks. In held-out Telecom tasks, the test evaluator’s mean completion reward increased from 29.50% with original tools to 36.75% with AT-AT tools, averaged across tasks and 30 trials per task.
A paired permutation test found the difference statistically significant. It supports an average gain in that setting, not a claim about every task or run.
- The control has a confound
A restricted version that could edit descriptions only performed substantially worse than full AT-AT in Airline and Telecom, while Retail showed no significant difference from the original tools. But this comparison changed two things: permission to edit code and whether traces appeared in the optimizer’s user prompt.
The result therefore does not isolate the effect of implementation edits. Separately, an LLM audit estimated a higher share of implementation-addressable failures in Telecom than Retail, though it classified only a small sample.
- Production outcomes varied
The production test checked a different setting: three Netflix servers, with AT-AT editing descriptions from earlier task traces. Held-out task pass rates rose on the Engineering Context server, stayed the same on Dev Tools, and remained zero on Collab Toolbox.
Tool errors and/or session costs fell on all three, but content scores also fell on two. The results show that outcomes varied by server and measure, rather than a uniform pass-rate increase.
Deep
- Original paper authors
- Maya Ravichandran, Pragya Tripathi, Dharamendra Kumar, Adith Swaminathan, Nathan Kallus
- Submitted to arXiv
- License
- CC BY 4.0