One distillation setup tolerated changes that broke reward training
How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate? · arXiv:2611.11450
In plain words
In on-policy distillation, a student learns from answers it generates itself, while a fixed teacher helps score those answers.
The paper measures how much the answer-generating engine differs from the model being trained, then uses that measure to check whether training still preserves agreement with the teacher.
For one 7B student trained on math, the tested setups with smaller differences kept teacher agreement within the study’s quality rule. Runs with larger differences crossed it, so the study gives a range for this student rather than one universal cutoff.
The research
A changed response engine can alter which partial answers a student learns from, so its effect on teacher agreement matters when judging a training setup.
This study gives that difference a common measure and a student-specific screen, while showing that levels tolerated by distillation did not carry over to the tested reward-based methods.
The study measures how far a response-generating engine’s token predictions differ from the trainer’s, then calibrates a check for on-policy distillation’s tolerated mismatch.
For each rollout setup, the researchers compared the trained student’s mean difference from its teacher on 530 held-out math problems with the standard BF16 run at the same step. A run failed when that difference rose at least 20% and a problem-paired interval excluded zero.
All tested setups through 0.184 nats per token passed, while runs near 0.39 failed with about a 23% rise. This places the observed transition in a bracket, rather than at a precise threshold.
The budget bracket depends on the chosen 20% quality threshold and applies to the tested 7B student and math training setup. Most configurations had one run, and the screening decisions were not tested near the admission boundary.
What the evidence shows
- A shared measure for OPD
The paper presents a measurement method and screening check for on-policy distillation (OPD), where a student learns from its own generated responses scored by a fixed teacher. The method asks how much the response-generating engine can differ from the model the trainer updates before teacher agreement worsens.
It gives different rollout changes a shared measure and checks candidate setups before full training.
- What counted as a failure
The researchers measured differences in token probabilities on the partial responses visited by each rollout engine, then averaged that measure across training steps. They judged quality by how far the trained student’s token probabilities were from the teacher across held-out math problems.
A run failed the primary rule only when that average rose at least 20% above the standard run and a paired statistical interval supported a difference.
- A bracket, not a cutoff
For the tested 7B student, every configuration at or below 0.184 nats per token passed the quality rule, while both tested noise runs near 0.39 failed. This brackets the observed transition under the study’s chosen threshold.
It supports substantial tolerance in this setup, but the gap between the passing and failing levels means the study did not identify an exact cutoff.
- Tolerance depends on the objective
The same rollout changes did not carry over to the tested reinforcement-learning methods, which trained from correctness rewards rather than teacher-scored token differences. REINFORCE broke in all three tested OPD-tolerated configurations, and GRPO broke in two of three.
Their standard-engine runs did not show measurable accuracy gains within 200 steps, so this contrast establishes failures under those tests, not relative learning gains.
- A calibrated screening check
A lightweight offline check matched all eight decisions fixed before full training, including five based on that check. It screened six configurations in 24 minutes on one GPU, but the student-specific line still required calibration from representative training runs.
Since no tested prediction was near the decision boundary, the results support the check as a screening aid rather than a guarantee.
Deep
- Original paper authors
- Cong Yu, Ziting Wang, Songlin Jiang, Sitong Zhang, Chengliang Chai
- Submitted to arXiv
- License
- arXiv non-exclusive