Distillation saves compute when it merges trained specialists
Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO · arXiv:2611.08453
In plain words
This paper compares two ways to train a language model using its own answers. In one, a frozen teacher scores each token the student writes.
In the other, training uses whether whole answers are right. The researchers count training compute and check accuracy on separate held-out math, code, knowledge and instruction-following tests.
With outside teachers, token-by-token distillation usually gets less accuracy for the compute spent. Its clearer win comes when it combines math and code specialists already trained with reward-based learning. That combined model matched joint training with less total compute.
The research
Choosing a training recipe by rollout count or step count can hide the cost of scoring every generated token.
For one task, a strong trained model may already be the desired result; the meaningful opportunity here is whether separate specialists can become one multi-domain model for less total compute than training them jointly.
The study compares on-policy distillation with reward-based training and finds its clearest total-compute advantage when it merges domain specialists into one model.
The study tracks held-out accuracy against training compute, comparing on-policy distillation (OPD), which matches student token probabilities to a frozen teacher, with Group Relative Policy Optimization (GRPO), which learns from answer outcomes. Main external-teacher comparisons share student initialization, prompts and rollout group size.
External-teacher OPD generally delivered less accuracy per compute. Distilling a student’s own GRPO checkpoint recovered its math accuracy with far less distillation compute, while combining math and code specialists matched joint training with lower total compute.
The single-domain compute comparison counts distillation after the GRPO-trained teacher already exists; it excludes the cost of producing that teacher. The findings concern vanilla reverse-KL distillation with a closed-form estimator over the student’s top 16 tokens.
What the evidence shows
- Compare compute, not steps
The paper compares on-policy distillation (OPD), which matches a student’s token probabilities to a frozen teacher, with Group Relative Policy Optimization (GRPO), which learns from relative answer outcomes. It tracks held-out accuracy against training FLOPs, the counted arithmetic operations during training.
OPD requires a teacher calculation for each generated token, so fewer rollouts or steps do not automatically mean less compute.
- External teachers usually lag
Across the main comparisons with external teachers and the reported tests that varied training factors, OPD generally delivered less accuracy per training FLOP than GRPO. The main comparisons shared student initialization, prompts, rollout group size and shared hyperparameters.
The pattern indicates that dense teacher feedback alone did not give this version of OPD a reliable compute advantage over outcome-based training.
- Weak rewards do not settle it
A difficulty study tested the idea that teacher feedback might help most when the student rarely solves a problem. Researchers formed eight math subsets using solve rates from a Qwen3-1.7B student and a Qwen3-4B thinking teacher.
GRPO’s final in-domain accuracy exceeded OPD’s peak in all eight. On held-out math, OPD finished higher in three subsets where the teacher solved the fewest prompts, while GRPO finished higher in the rest.
- A trained checkpoint transfers cheaply
Using a GRPO-trained checkpoint as teacher changes the single-domain comparison: on held-out math, distillation with one rollout per prompt matched the teacher’s mean accuracy across 16 responses per problem with 0.2 EFLOP of distillation compute, while direct GRPO needed 10.1 EFLOP to reach that accuracy.
The reduction applies to distillation compute only; it excludes the GRPO training needed to produce the teacher.
- Combining specialists changes the cost
For two domains, the researchers trained separate math and code specialists, then distilled them into one student. Counting both specialist training and distillation, the pipeline matched joint GRPO training on that mix with 42.0% fewer total FLOPs.
On a separate four-domain mix of math, code, instruction following, and knowledge, it also matched joint GRPO instance-mix performance with 31.6% fewer total FLOPs.
Deep
- Original paper authors
- Zeyu Leo Liu, Yang Li, Bo Pang, Emre Can Acikgoz, Xuan-Phi Nguyen
- Submitted to arXiv
- License
- CC BY 4.0