Distillation can miss abilities beyond its training examples
On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation · arXiv:2611.05609
In plain words
A model can learn to copy a teacher’s answers on the tasks it sees and still struggle when a task goes beyond those examples. This paper studies on-policy distillation, where a student learns from its own answers with a teacher’s guidance, and a version that combines several teachers.
In a small synthetic math test, the student picked up nearly all the teacher’s gains on familiar problem ranges, but only 76–86% of those gains on harder problems requiring 15–17 operations. The result is one example of why strong familiar-task scores may not show that every ability has transferred.
The research
A student can look successful on the tasks used to train it while losing ground on harder tasks or failing to combine separately taught abilities.
These results make familiar-task scores an incomplete check of consolidation: transfer and combination need to be examined directly, and a strong teacher alone may not be enough under narrow supervision.
The paper tests whether on-policy distillation (OPD) and its multi-teacher extension (MOPD) preserve abilities beyond supervised tasks or combine them.
The authors trained Qwen3 students with one or several teachers, then compared their scores with the base model and teachers on math benchmarks the paper grouped as in-distribution (IND) when each of three self-reinforcement-learning teachers scored above 97% of the base, and grouped the remaining benchmarks as out-of-distribution (OOD), and on tasks combining separately supervised capabilities. A stronger external teacher alone did not prevent below-base math scores under narrow supervision.
MOPD reduced average math forgetting relative to mixed reinforcement learning, but remained below the base on the OOD group.
Across three composition settings, its average composed-task score also fell short of the best teachers, showing that gains on component tasks did not ensure gains when capabilities had to work together.
The main post-training experiments use Qwen3-8B students, public data and tests, and limited tuning of the combined methods.
The paper’s cross-benchmark math OOD labels are based on teacher performance relative to the base, not on verified training-data membership; evaluation sampling checks do not measure variation across independent training runs.
What the evidence shows
- Strong scores can hide gaps
Multi-teacher on-policy distillation (MOPD) is a method for training one student from several teachers assigned to different task domains. The paper asks whether strong performance on supervised tasks means the student has retained abilities beyond those tasks or can use separately taught capabilities together. It tests both questions through mathematical extrapolation and tasks that combine capabilities.
- Transfer fell on harder problems
In this synthetic reasoning test, the student captured 93–96% of the teacher’s improvement on the post-training range but only 76–86% on harder problems. Strong transfer on supervised problems therefore did not guarantee complete transfer to the tested harder range.
- A stronger teacher was not enough
Even with an external Qwen3-32B teacher, narrow math supervision left the Qwen3-8B student’s mean score 2.96 percentage points below its base on tests the paper grouped as outside the supervised range using teacher performance. This result shows that a stronger teacher alone did not ensure retention in this setting.
- Less forgetting, incomplete combination
Across the paper’s math tests classified as outside supervision by self-trained teachers’ scores relative to the base, MOPD reduced average forgetting versus mixed reinforcement learning but remained below the base itself.
This comparison shows a relative improvement over that method, not full retention in the tested mixtures.
- Broader data changed the result
A data-coverage comparison points to a possible way to change the math outcome. With the Qwen3-32B teacher held constant, adding 10% OpenR1 to GSM8K training coincided with a Qwen3-8B student scoring above its base on all five reported math tests.
This bounded result supports considering teacher strength and training-data coverage together; it does not show that broader data will have the same effect in other settings.
Deep
- Original paper authors
- Yichen Wang, Yufa Zhou, Siqi Zhu, Haozheng Luo, Jingzhi Bao
- Submitted to arXiv
- License
- arXiv non-exclusive