More varied tool training data helps as budgets grow
Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis · arXiv:2610.37648
In plain words
Agents that use tools must choose the right tools over several steps and keep track of changing information. Realistic examples for training them are scarce.
SynthAPI creates small executable apps with database-backed tools, then makes tasks with starting data and checks that confirm whether each task was solved. Researchers found that as training sets grew, adding more apps and different starting data often helped more than simply adding tasks to familiar apps. The result comes from simulated tests, not live users.
The research
When training examples are limited, researchers face a practical question: should they spend a fixed task budget on more tools, more tasks for familiar tools, or more varied starting data?
Earlier environment-generation work emphasized quantity, leaving unclear how the mix of examples affects agents’ ability to handle new tools and environments.
SynthAPI lets researchers vary generated tool environments, then uses those comparisons to predict which mix of apps and database states best supports agent training.
SynthAPI generates executable, database-backed tool apps and tasks so researchers can vary the training data mix. Its scaling law predicts downstream scores using that mix, and the researchers compared it with a law based on task count alone.
On runs in the top quarter of training compute, held out from fitting, the composition-aware law’s mean absolute prediction error was 0.0285 normalized-score points, versus 0.1706 for the task-count-only law.
The score scale runs from each starting model’s no-SynthAPI result to the highest observed trained result on each test.
The database-complexity result may partly reflect that the downstream tests were not sensitive to harder database states. SynthAPI does not simulate users answering agents’ questions, and its scaling law has only been validated on this pipeline.
What the evidence shows
- A data composition question
SynthAPI is a pipeline for generating executable environments and verifiable tasks to train and study agents that use tools over multiple steps. Those agents need to choose and combine tools while tracking changes to information.
Because such examples are scarce in ordinary internet text, the researchers use SynthAPI to study which parts of generated training data matter as the task budget grows.
- Making the data controllable
The pipeline creates Python apps whose tools read or change structured databases. It generates task instructions, starting database states, success checks, and solver programs that verify the tasks are feasible.
Researchers can vary the number of apps, tasks per app, database states per instruction, and extra data in each state, making different data compositions comparable at a fixed task budget.
- Measuring downstream success
For the diagnostic analysis, AppWorld tests device-assistant tasks that combine tools and information across apps; SynthAPI-Holdout tests tasks in newly generated apps with new instructions and database states. Researchers measured task success over four trials per episode and combined normalized scores across these two tests.
Each normalized score runs from the starting model’s no-SynthAPI result at zero to the highest result observed among trained models at one.
- The best mix shifts
The useful mix changed with training budget. For the 9B model, eight apps performed best at 64–512 tasks, while 64 apps performed best from 4,096 tasks; the fitted performance ceiling also rose with app count.
More starting database states per instruction likewise performed best at larger budgets. At high budgets, the largest distractor-data setting improved diagnostic scores, although the authors say these tests may be insensitive to harder database states.
- Predicting and checking recipes
A scaling law that used app count, task allocation, database-state variety, and distractor data predicted higher-compute results more closely than one using task count alone. A separate fixed-budget comparison found that the more diverse SynthAPI recipe scored higher on four downstream tests across the three reported model sizes.
Together, these results support accounting for data composition when choosing a recipe within SynthAPI, while leaving its transfer to other pipelines untested.
Deep
- Original paper authors
- Andrew Szot, Samira Abnar, Omar Attia, Abbas Kazerouni, Michael Kirchhof
- Submitted to arXiv
- License
- arXiv non-exclusive