Choosing fewer, better tokens improves distillation

If a student model receives a training signal for every token it generates, does that extra supervision always help? Sampled-token distillation measures how much the student and teacher disagree, but that score can be large even when both models assign a token very low probability. The paper tests whether choosing tokens by both disagreement and probability improves learning.

DIAL-OPD ranks each student-generated response token by disagreement adjusted for probability, then trains on only the highest-scoring fraction.

Researchers tested whether a student distilling from a teacher needs training signals for every token it generates. The usual sampled-token reward measures disagreement but ignores how probable the token is under both models, so a large reward can go to a token both consider unlikely.

DIAL-OPD ranks tokens using disagreement adjusted for probability and trains on a chosen fraction. With the same Qwen3 student start, teacher, data, loss, and update budget, random selection at 80% retention reached 38.4% accuracy on MATH-500, a test of mathematical problem solving, versus 30.2% for full-token Vanilla OPD. This suggests fewer training tokens can help in that setup.

The evidence comes from Qwen3 teacher-student pairs trained on DeepScaleR and evaluated on seven mathematical reasoning tests. It does not establish that the results hold for other model families, training data, or tasks.