Matching training optimizers adds little consistent benefit

When a language model is trained in stages, does changing the method that updates its parameters between stages undermine its later task performance?

The answer matters to understanding whether a later training stage needs to match the method used to build the model, or whether each stage's choice can be assessed on its own.

The authors varied AdamW and Muon separately across training stages and found little consistent downstream benefit from matching them.

The authors trained 1.2B-parameter Llama-3 models with different AdamW and Muon mixtures in pretraining and task-general fine-tuning. They compared six downstream test averages across the pairings, then separated each optimizer's average contribution from the remaining interaction between stages.

Those interactions were small and inconsistent, generally comparable to variation across three fine-tuning runs. More Muon updates during fine-tuning generally raised the test average, regardless of the pretraining mixture.

A separate measure found that matching mixtures reduced forgetting of pretraining data, but this did not raise downstream scores.

The main optimizer comparison uses 1.2B-parameter Llama-3 models and six downstream tests. The open-weight comparison covers five named Llama and Qwen settings; neither establishes the same result for other optimizers, model families, or training stages.