In plain words
The paper introduces Rollout-to-Scorer Distillation, a way to improve the advice a teacher gives a student agent. The teacher first takes over from selected points in the student’s attempts.
A copy of the teacher learns scores informed by those takeovers, then scores the student’s words during training without showing the student the takeover.
In tests across three text-based task environments, students trained with this repaired scorer completed tasks more often than students trained with four comparison methods. The result is limited to the tested models and environments.
The research
When a student spends turns searching or checking, can a teacher’s token-by-token advice tell it which moves will actually help finish?
In ordinary agentic distillation, the teacher scores the student’s words without seeing the episode’s ending, so locally sensible delay can pass without an outcome penalty.
Rollout-to-Scorer Distillation uses selected teacher takeovers to repair a teacher’s scores, then freezes the repaired scorer for ordinary student training.
To test the mismatch, the authors compared teacher takeovers with student outcomes from the same high-disagreement states in three text tasks: household activities, shopping, and science experiments. The teacher finished more often, suggesting those states remained within its ability to handle.
Evaluators averaged task success over three sampled decodes of each final checkpoint. R2S scored higher than ordinary distillation and three recent methods in all twelve teacher–student–environment cells, supporting the repair within the tested combinations.
The main comparison used one 150-step training run per model–environment cell, and the comparison methods retained their own settings. The evidence covers the tested Qwen models and replayable text environments; it does not establish results beyond them.
Analysis
Deep
- Submitted to arXiv
- License
- CC BY 4.0