In plain words
Computer-use agents interact with software through tools that let them see and act on a computer. Those tools are often fixed by people, even when a task might call for a different way to work.
Agent S4 is a method that lets a separate software agent improve a reusable set of tools and instructions by studying task attempts.
On a test of scientific software tasks, the evolved set helped the same model complete 99 of 169 tasks, compared with 93 using the starting set. This shows a modest improvement in that evaluation, not a guarantee for other software or tasks.
The research
When an agent’s tools are fixed in advance, they may not fit the task, and adding new tools usually takes manual work. Can an agent improve a reusable set of interfaces from its own task attempts, while keeping the task-solving model and harness fixed during that search?
Agent S4 evolves reusable computer-use interfaces and instructions from task attempts, then uses separate validation scores to select a repository.
On a full test of computer-use tasks, the evolved setup completed more tasks than the same model using fixed interfaces. The comparison also changed the harness and execution access, so it cannot isolate the repository’s effect.
In a separate test of scientific software tasks, the evolved repository solved 99 of 169 tasks, versus 93 with the seed repository. Because model and harness matched, this supports a repository improvement in that setting.
The OSWorld V2 comparison also changed the harness and execution access, so it cannot show how much of the gain came from the repository alone. The selected task examples illustrate possible mechanisms but are not controlled comparisons.
Analysis
Deep
- Original paper authors
- Saaket Agashe, Yuzhe Yang, Gonzalo Gonzalez-Pumariega, Bhuvan Chandra Koduru, Chih-Lun Li
- Submitted to arXiv
- License
- CC BY 4.0