Near-one momentum improved validation accuracy in two tests

When earlier experiments found that momentum could fail to help or even hurt reward-based language-model training, it was unclear whether averaging gradients was ineffective or whether the usual setting was poorly matched to the learning rate. The paper asks when that averaging can produce a visible gain.

The paper combines bandit analysis and language-model tests to show when longer-lasting gradient averages can improve reward-based training.

The authors compare momentum settings in a simplified response-choice model, then analyze what changes when gradients come from sampled responses. With a fixed setting, the expected gain over no momentum shrinks faster than run-to-run fluctuations, so one sampled run is not reliably better.

Scaling the setting with the learning rate avoids that vanishing gain in the analysis. In a Qwen2.5-32B test, an answer verifier’s scores averaged 46.1% across 8,000 validation responses at step 150, versus 44.3% with the conventional setting.

The formal analysis uses a response-level softmax bandit with deterministic binary rewards and a penalty for moving away from a reference policy.

The language-model tests cover two models, one training dataset, and three training seeds per configuration, so results beyond those conditions remain unestablished.