COPC corrects stale weights and learning signals in language-model training
In plain words
When a language model is trained asynchronously, it may learn from answers produced by an older version of itself. Even if training adjusts how much those answers count, the estimate of whether an action was useful can still rely on what the older version would have done next.
COPC is a training method that corrects both the token updates and those estimates. In the reported tool-based math setting, its average accuracy across five tests was higher than AReaL's; each problem had 32 sampled answers.
The research
When training overlaps response generation with model updates, can adjusting the weight of older responses fix the learning signal too? The advantage estimate can still depend on what an older version would have generated next, so policy-side correction alone may leave a mismatch.
COPC is an actor–critic method that masks stale token updates and separately corrects return estimates used to train the critic and calculate actor advantages.
The authors compare COPC with AReaL from the same supervised-trained backbone in each task, though their reinforcement-learning methods and settings differ. COPC masks some token updates and corrects return estimates that shape critic training and actor advantages.
On tool-integrated math problems, average accuracy across five tests was 50.29 for COPC and 48.26 for AReaL; each problem had 32 sampled answers. The higher result supports COPC in this setting.
The fixed-point result assumes a finite horizon and that actions the learner may take are also possible under the older policy. The experiments cover two task settings and two model backbones; the analysis does not give a rule for choosing the best correction strengths.
Analysis
Deep
- Authors
- Zicheng Hu, Zhijian Zhou, Xuan Zhang, Yuchen Liu, Cheng Chen
- License
- arXiv non-exclusive