Language models preserve useful past predictions with FSPO
In plain words
Function-Space Policy Optimization (FSPO) is a method for training language models on reasoning and tool-use tasks. It learns from a model’s current answers and from selected predictions saved from earlier training, so those past targets do not change every time the model updates.
The authors also strengthen reference-solution guidance when a model keeps failing a problem and reduce it as the model solves problems reliably. In tests on two models, FSPO had higher average accuracy than on-policy SDPO across five tasks after five hours; one-hour Tool Use results favored SDPO.
The research
Can a language model learn detailed guidance from its own answers without letting that guidance shift with every update or losing useful earlier predictions on reasoning and tool-use problems? The paper asks whether training can preserve those past signals while still adapting to new evidence.
FSPO trains one language model from current privileged examples and saved sparse predictions from earlier states, preserving past guidance without a separate teacher model.
FSPO pairs each sampled answer with stronger evidence, such as a verified sibling answer or fallback reference solution, and trains the model to match the resulting next-token guidance.
It also replays saved predictions from earlier model states as fixed targets, while weighting reference guidance more when the model’s recent solutions are unreliable.
In held-out Chemistry, Physics, Biology, Materials, and structured Tool Use tests, FSPO’s average accuracy across five tasks was higher than on-policy SDPO for both tested models under a five-hour budget. This supports combining current and historical guidance in the tested setting.
The results cover two language-model backbones, five held-out reasoning and tool-use tasks, and the reported training budgets. On Tool Use at one hour, on-policy SDPO scored higher than FSPO on both models; the ablation and memory estimates also apply only to their stated settings.
Analysis
Deep
- Original paper authors
- Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma
- Submitted to arXiv
- License
- CC BY 4.0