Match each distillation choice to its measured outcome
A team distilling a model can supervise fewer generated tokens, change who generates its training responses, or ask it to produce an answer in fewer decoding steps.
Each choice changes a different part of the process, so a quality gain under one choice does not establish a shared saving in computation or stored memory.
The studies offer a practical starting rule: name the choice, the behavior being measured, and the comparison before interpreting a result. Their reported gains belong to particular tasks and protocols.
What you will understand
Which choice changes supervised tokens, training responses, or decoding steps?
What behavior did each study measure, and against which comparison?
Which limits prevent a result from becoming a general decision rule?
Identify the choice before judging the result
Imagine training a smaller model to solve a math problem from a stronger teacher. The student writes its own answer, and the teacher supplies feedback on the generated words and symbols, called tokens.
Applying that feedback at every token sounds thorough. Yet a sampled-token training method receives only a noisy glimpse of the teacher’s preferences at each position.
The question for this team is whether every available bit of feedback helps the student learn to reason, or whether some positions deserve more attention.
A second choice concerns the response used for training. The team can repeatedly ask the current student to generate responses, then consult the teacher along those responses.
That is on-policy distillation. It can also train on responses generated by the teacher, the off-policy option tested in a controlled study.
The distinction changes which partial answers the student encounters during training. By itself, it says nothing about how the teacher’s predictions are compared with the student’s, or whether the student will retain earlier abilities.
Now suppose the model must also caption an image or generate one. In the multimodal study, it starts with masked positions and fills them over several decoding steps.
Asking it to finish in fewer steps means committing more positions together; for text, those positions cannot use one another’s newly chosen words. These three situations pose one shared decision problem: identify the training or decoding choice being changed, then ask which outcome was actually measured under that choice.
Where the lesson goes next
Choose which generated tokens receive supervision
Test where training responses come from
Shorten decoding while checking the output
Keep the decision tied to its test