Multi-turn RL gains depend on interaction budgets
What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents? · arXiv:2611.06483
In plain words
The paper asks what multi-turn reinforcement learning teaches language models that act through a series of steps. It finds that training can help a model succeed across more sampled attempts, but the advantage often shrinks when each attempt has fewer interactions.
The researchers also find that trained models are not consistently better at the next step, but can do better at continuing after selected errors.
A training strategy called BudgetMix varies the interaction limit during training; on four tasks, it improves results at tighter limits while matching the usual approach at the full limit.
The research
A high task-completion rate can make an agent seem more capable without showing whether it can make better decisions at each step or work within a tight interaction limit.
This analysis changes what those success rates can support: evidence of improvement across sampled attempts does not guarantee an advantage when each attempt has less room to act.
The paper analyzes what multi-turn reinforcement learning changes in language-model agents and introduces BudgetMix RL, which varies interaction limits during training.
The researchers compare base and RL-trained models by varying the number of sampled attempts and, separately, the maximum interactions per attempt. More attempts reveal an RL advantage, but with eight attempts fixed, that advantage often weakens under tighter limits.
At matched states, RL is not consistently better at the next action; from selected error states, its continuations succeed more often. These comparisons support the interpretation that recovery and longer execution matter more than uniform step-level improvement. BudgetMix then tests training across interaction limits.
The action-quality and recovery tests cover two tasks, and their error states were constructed as proxies. The budget pattern also varies by task, so it does not describe every agent or setting.
What the evidence shows
- Success has two dimensions
The paper’s analysis of multi-turn reinforcement learning (RL) asks what task success alone misses. A model may succeed more often when given more sampled attempts, while still failing to make better decisions at each step or to retain its advantage when each attempt has fewer interactions.
The paper therefore examines both the number of attempts and the interaction limit within each attempt.
- More attempts, longer paths
In the main comparison, the RL-trained models outperform their base models as the number of sampled attempts increases. But with eight attempts held fixed, the advantage generally grows with the interaction limit; on some tasks it shrinks, disappears or reverses under tighter limits.
This means that a higher success rate across attempts does not by itself show that the model is better suited to every interaction budget.
- Next actions are not uniformly better
To test whether RL improves each decision, the researchers compare both models at the same intermediate states. RL has higher measured action quality at 46.8% of matched coding-task states and 42.8% of matched navigation-task states, while its mean quality is no better overall.
The comparison weighs against a consistent next-action improvement as the explanation for the overall success gains.
- Recovery helps explain the gap
The recovery comparison points to another difference: from the same constructed error states, RL-trained continuations succeed more often than base-model continuations in both tested tasks.
Together with the budget sweep, this supports the authors’ interpretation that RL gains are associated more with recovering from selected errors and sustaining longer attempts than with uniformly better individual actions.
- Training across budgets
BudgetMix varies interaction limits during training while keeping the corresponding full-budget method’s reward mechanism.
In the reported four-task evaluation, this comparison suggests that training across limits can improve performance under tighter interaction limits while preserving performance at the full limit.
Deep
- Original paper authors
- Kejian Zhu, Zhuoran Jin, Shangqing Tu, Tianyi Men, Yupu Hao
- Submitted to arXiv
- License
- CC BY 4.0