EN
A scheduler uses tool timing to place waiting AI sessions
TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving · arXiv:2610.37230
In plain words
AI agents take turns asking a language model for help and waiting for outside tools, such as a compiler. While one agent waits, its GPU time can go to another.
The paper introduces TAO, a scheduler that uses when tools may return, reusable conversation data, and session size to choose which waiting agents use which GPUs.
Across 12 reported model, workload, and concurrency settings, TAO's largest reported reduction in average task time was 26.8% versus the best tested scheduler for that measure. That task time includes tool execution, and the largest gain did not occur in every setting.
The research
When an AI agent waits for an outside tool, its GPU capacity may be available to another session, but a scheduler must also account for when the waiting session will return.
How should it choose among sessions that compete for the same GPU nodes while preserving useful cache data and serving those likely to need the model soon?
TAO combines tool-return timing, cache reuse, and session size to choose and place waiting agent sessions under GPU capacity limits.
TAO compares its scheduler with evaluated alternatives across 12 model, workload, and concurrency settings. The largest reported reductions were 26.8% in mean task latency and 36.3% in mean time to first token; decode throughput rose by as much as 29.5%.
Task latency includes tool execution, and each maximum is relative to the best evaluated baseline for that measure and setting.
These results support the claim that TAO can improve serving performance in tested conditions, but the gains are not uniform. At 128 workers on Qwen3-32B with SWE-verified, its mean task latency was 2.5% higher than ThunderAgent's.
The reported gains are maximum improvements across 12 model, workload, and concurrency settings, not results for every setting.
The paper's bound for plans applied after a snapshot concerns scheduling value under stated conditions; it does not bound task latency or performance against a globally optimal online scheduler.
What the evidence shows
- The scheduling question
TAO is a tool-aware joint scheduling framework for language-model agent serving. Agent sessions alternate between requests to a model on clustered GPUs and calls to external tools on distributed CPUs.
When a tool is running, its session does not need GPU inference, so the scheduler has an opportunity to allocate that capacity elsewhere. The question is how to account for the session's likely return while choosing among several sessions competing for the same nodes.
- Why joint choices matter
Existing schedulers can overlook tool-return timing and make admission and placement choices with per-session heuristics. But sessions compete for shared node capacity: choosing one can rule out a more valuable combination of others.
TAO gives each eligible session–node pair a value based on cache credit, near-term tool-return readiness, and token count, in that priority order, then chooses assignments jointly under node capacity limits.
- What the main results show
The paper compares TAO with evaluated schedulers across 12 model, workload, and concurrency settings. It reports maximum relative improvements of 26.8% in mean end-to-end task latency, 36.3% in mean time to first token, and 29.5% in decode throughput.
Task latency includes model inference and tool execution; throughput counts output tokens per second. Each maximum is relative to the best evaluated baseline for that metric and setting, which can vary.
- A setting with higher latency
The maxima are not gains in every setting. In one reported Qwen3-32B comparison on SWE-verified at 128 workers, TAO's mean task latency was 2.5% higher than ThunderAgent's. This exception makes the scope of the headline results clear: the paper reports best relative gains across a set of settings, not a guarantee that TAO reduces latency in each one.
- What the component test adds
A separate component comparison used DeepSeek-V4 Flash on SWE-Lite at 32 workers across two nodes. Complete TAO produced 1,417.97 output tokens per second, while variants that removed cache value or readiness, changed pause ordering, or replaced joint resumption had lower throughput.
The comparison supports a contribution from both valuation and resumption decisions in that setup, while its single-setting scope limits how broadly those component effects can be read.
Deep
- Original paper authors
- Shuhuai Li, Shengdong Ye, Dongdong Ge, Yinyu Ye, Jianghao Lin
- Submitted to arXiv
- License
- CC BY 4.0