A token-wise update keeps stale-response training aimed at its target

When a language model learns from answers generated by an older version of itself, correcting each sampled word can reduce the instability of correcting a whole response, but may also shift the update away from its intended target. Can an update use word-level correction and still retain that target?

Proximal KL pairs each sampled token’s correction for the policy that generated it with bounded future costs from a fixed reference, to preserve the update’s target.

Can token-by-token correction train a language model from older responses without losing the target of a KL-regularized update? Proximal KL combines that correction with bounded costs from future tokens, measured against a fixed reference.

Researchers compared its trained models with GRPO on eight held-out math test sets, which check whether generated answers are correct. Proximal KL had higher response-level aggregate accuracy in all seven matched off-policy comparisons, supporting an advantage in the tested settings.

The stationary-target theorem assumes an exact population update with full future KL credit. The practical update clips or truncates costs and uses group-normalized rewards and minibatch token averages, which can change its expected update.

The accuracy comparisons cover two Qwen3 sizes and the reported math-test suite, not other models or tasks.