Matching compute changes which training methods pay off
Are Dense Rewards Worth Their Tokens in GRPO? · arXiv:2611.13560
In plain words
Some language-model training methods spend extra computing to give more detailed credit for each step, while others spend it choosing which problems to practice. The paper introduces a way to compare these choices using equal budgets of generated text and training time.
In the tests, extra step-by-step credit looked helpful when both methods took the same number of updates, but fell behind when its extra generated text counted against the budget. Choosing prompts helped on two tested models, though not across all six.
The research
Training methods can spend computing on richer feedback or on choosing more useful practice problems, and update counts alone can make the first choice look better.
Counting generated text and training time changes that comparison: in these tests, the apparent lead for extra step-by-step credit disappeared, while prompt selection helped in two model settings.
The paper introduces a protocol that matches training interventions by generated tokens and GPU time, then tests dense rewards and prompt selection in GRPO.
The protocol compares interventions under equal generated-token and training-time budgets, then measures sampled-answer accuracy on held-out numeric-answer MATH problems. On Qwen2.5-1.5B, a dense reward led after equal updates but fell behind when its continuation tokens counted toward the same budget.
Prompt selection improved accuracy on two tested models, connecting the gains to a different use of compute.
These comparisons cover small models and numeric-answer MATH training under the tested budgets and recipes. The results do not settle how the interventions behave at larger scales, on other tasks, or with learning rates tuned separately for each model and setup.
What the evidence shows
- Compare the full compute cost
The paper introduces a matched-compute protocol for comparing interventions that change language-model training. It counts both generated tokens and training GPU-hours, so a method cannot appear cheaper merely because scored samples or other training stages were left out.
The authors apply it to two ways of spending compute in group-relative policy optimization: giving credit to intermediate solution steps and choosing which prompts to train on.
- Update counts reverse the result
With the same training updates, the extra step-by-step reward looks better, but counting its extra generated text reverses the comparison. This shows that update counts alone can make a more costly method appear to improve accuracy.
- Prompt choice helped in two cases
The filter’s average accuracy gains appeared on two tested models in the MATH levels 3–5 studies. This suggests that selecting prompts can help in some tested settings, but does not establish a gain across models.
- Controls test the selection account
On Qwen2.5-1.5B, a control choosing randomly from the same candidates scored lower than the filter, supporting a role for which prompts were selected in that comparison.
On Qwen3-0.6B, shifting the target toward harder prompts removed the gain over ordinary training, suggesting that the selection direction mattered in that tested setting.
- The result stays within MATH
The filter’s gain persisted on further test problems from the same MATH distribution, extending the comparison beyond the repeatedly used test set. The result remains specific to the tested models, numeric-answer tasks, budgets and training recipes.
Deep
- Original paper authors
- Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri
- Submitted to arXiv
- License
- CC BY 4.0