An optimizer change slows forgetting during adaptation

When a language model learns from a new corpus, it can become worse at older subjects. Replay can help preserve them, but it requires the original training data, which may not be available. The paper asks whether adaptation can limit those losses without access to that data.

The method raises AdamW’s epsilon only for output token embeddings during training, damping their drift without access to the original data.

The researchers changed one optimizer setting during Korean adaptation: they increased AdamW’s epsilon only for the output projection, which produces scores for the next token. The change slows updates to output token embeddings that the new corpus rarely uses.

Compared with matched training runs without the change, the average increase in old-domain validation loss fell by 39.4%–67.9% across seven stable settings. This supports the adjustment for the tested settings; it does not show that every earlier capability is fully preserved.

The main result covers seven stable settings in an eight-setting grid. The excluded Pythia-160M peak-rate run collapsed; in a six-seed check at Pythia-410M, target learning fell by 7.0%, within the reference runs’ 18.9% spread. The aggregate does not describe every run.