Twice gives search agents credit for individual turns

When a language-model search agent receives reward only for whether its final answer is right, which query or stopping choice deserves credit? This matters for training because rewarding every turn alike can punish a helpful query in a failed episode and reward a wasted query in a lucky episode.

Twice estimates each search decision’s credit by changing that turn in a paired episode while reusing the original episode’s later sampling noise.

For each model, researchers averaged exact-answer scores across seven question-answering tests and three training seeds. Twice scored higher than GRPO, with averages 5.3 points higher for Qwen2.5-3B and 4.5 higher for Qwen2.5-7B.

The methods shared training data, retrieval, reward, GRPO loss, and update count. These results support higher average scores in the tested setup, consistent with the idea that comparing paired turns can give training a more informative signal.

The results come from replayable-tool training and question-answering and two text environments, with models up to 14B parameters and at most eight search calls. Longer research tasks remain untested, and the main comparison does not specify whether starting checkpoints matched.