Info-GRPO links reasoning paths to training-time seeds
Correlation-Aware Exploration for Long-Trace Reasoning · arXiv:2610.34949
In plain words
Language models can settle on a narrow set of ways to solve long reasoning problems. Adding randomness to every next word may help them explore, but it can also push them toward unlikely words again and again.
Info-GRPO is a training framework that uses digital seeds added to prompts to guide different reasoning paths for the same prompt. The authors found higher average accuracy than GRPO on one mathematics test for one model, while also reporting a case where it did worse.
The result is promising evidence for this training approach, not a guarantee across models or tasks.
The research
Can training encourage a language model to explore different ways through a long reasoning problem without making each next word less predictable?
The authors argue that token-level entropy bonuses can push probability toward unlikely words throughout a long trace, creating tension between exploration and convergence.
Info-GRPO uses training-time digital seeds to link different reasoning paths with different seeds while each conditioned path becomes more confident.
Info-GRPO trains on original prompts and prompts augmented with digital seeds. Its objective encourages variation across unseeded paths while reducing uncertainty in paths conditioned on a seed; inference uses the original prompt.
On AIME24, a mathematics test, the authors report 62.08% Avg@8 for Info-GRPO and 57.92% for GRPO on DeepSeek-R1-Distill-Qwen-7B. Avg@8 is mean accuracy across eight independent decoding trials. This supports a gain in that comparison, not a universal advantage.
The results cover mathematics and code tasks with verifiable rewards, at response budgets up to 8,192 tokens.
The authors report no dispersion estimates for Avg@8 and did not repeat full training with independent random seeds, so the results do not establish how reliably gains would recur across training runs.
What the evidence shows
- The exploration problem
Info-GRPO is a training framework for language models that extends Group Relative Policy Optimization (GRPO). It addresses a problem in long reasoning traces: GRPO can reinforce high-reward paths and narrow the strategies a model uses.
The authors examine whether adding a token-level entropy bonus can encourage exploration without making training less stable.
- Testing the entropy bonus
A token-level entropy bonus spreads probability across possible next tokens. The authors argue that in a large vocabulary it places strong pressure on very unlikely tokens at every step, and that pressure accumulates over a long trace.
In an ablation on DeepSeek-R1-Distill-Qwen-7B, a full-vocabulary bonus scored lower on AIME24 than GRPO without a bonus, while restricting the bonus to head tokens scored slightly higher. This comparison tests the bonus’s scope in those three configurations.
- Linking seeds and paths
Info-GRPO adds digital seeds to prompts during training and combines generations from seeded and original prompts in its objective. The goal is for seeds to guide different reasoning paths while each seed-conditioned model becomes confident.
The seed is not added at inference. On AIME24, a mathematics test, Info-GRPO scored 62.08% Avg@8 versus 57.92% for GRPO on DeepSeek-R1-Distill-Qwen-7B; Avg@8 is the mean accuracy across eight independent decoding trials.
- Checking the seed control
A control helps distinguish the full method from prompt augmentation alone. On AIME24, GRPO trained with seed-augmented prompts but without the mutual-information regularizer scored 42.23% Avg@8, below vanilla GRPO’s 45.04% on Qwen2.5-Math-7B.
The control suggests that adding seeds by itself did not reproduce the reported gain, though it does not isolate every effect of seeds in Info-GRPO.
- Scope and exception
The reported evidence also includes a code test and an inference-time sampling comparison. On LiveCodeBench, which tests code generation, Info-GRPO had higher Avg@8 accuracy than GRPO for DeepSeek-R1-Distill-Qwen-7B.
On AIME24, that model reached 80% majority-vote accuracy with eight samples per question under Info-GRPO, compared with 64 for GRPO. Yet on AIME25, Info-GRPO scored below GRPO for Qwen2.5-7B, so the gains are not universal.
Deep
- Original paper authors
- Junxing Hu, Zicheng Zhang, Zhiqian Zhang, Ai Han, Yifeng Zhang
- Submitted to arXiv
- License
- arXiv non-exclusive