A harness-editing method improves transfer to unseen tasks
In plain words
An AI agent has a language model plus surrounding instructions, tools, memory, and rules for how it works. This paper introduces a method that changes those surrounding parts while keeping the model fixed. It tries to avoid changes that merely fit the tasks used during improvement.
The researchers tested the method by improving an agent on one set of workspace tasks and then testing it on three unseen sets. Its average score was higher than the score of the unchanged agent, though that average does not mean every task improved.
The research
When an agent is repeatedly changed using feedback from a finite set of tasks, does it learn changes that help on unseen tasks, or does it simply fit the set it was improved on? That distinction matters to whether measured improvement carries beyond the tasks guiding each round of changes.
RRSI keeps the language model fixed while regulating how an agent’s prompts, tools, and other harness parts are changed and retained.
The researchers evolved an agent harness on one suite in each domain, then ran it unchanged on held-out tests, with the backbone model frozen. In the agentic-workspace setting, the average score across three unseen tests was 43.6 for RRSI and 39.7 for the unevolved harness.
Because the average combines scores across those tests, it supports a transfer gain for that set rather than improvement on every task. The comparison tests whether the evolved harness helps beyond the tasks used to change it.
The study changes harnesses while keeping the backbone model frozen. It relies on a finite evolve set and regularization settings; transfer to substantially different agent architectures, tool ecosystems, and longer runs remains to be tested.
Analysis
Deep
- Authors
- Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhuang
- License
- arXiv non-exclusive