Why larger rollouts learn slower but fail on a similar schedule

Generating fresh model responses for every policy update makes training with checkable rewards costly.

Larger batches can spread that generation across more updates, but in multi-step training it is unclear whether the usual square-root learning-rate rule still preserves learning progress, or how changing batch size affects when training fails.

The authors’ two-clock framework separates responses needed to learn from responses before failure, explaining why larger rollouts slow progress but may not move the failure point.

The authors introduce a two-clock framework to ask whether larger response batches preserve sample efficiency in multi-step Group Relative Policy Optimization (GRPO), a way to train models from checkable answers.

They compare training runs at different rollout sizes under square-root learning-rate scaling, measuring responses consumed to reach validation-accuracy targets.

With eight optimizer steps per rollout, the sample-cost exponent rose from 0.15 to 0.44 as the small-batch learning rate increased from 10⁻⁶ to 10⁻⁵.

The larger batches therefore needed more responses to reach the same accuracy, while a separate fit to unclipped runs supported roughly batch-invariant failure counts along the scaling rule.

The evidence comes from multi-step GRPO training in one model family and one task domain. The theory is local, curvature was not directly measured at this scale, failure points varied across seeds, and efficiency was counted in responses rather than tokens, computing cost, or elapsed time.