Synthetic training fails to beat text on fixed budgets
Rethinking Pre-pretraining · arXiv:2610.40051
In plain words
When natural-language text is scarce, a model can spend extra training time repeating that text or first learning from made-up examples. This paper tests whether that early synthetic training helps the model learn natural language.
The authors find that some synthetic training helps when its compute is added, but the gain depends on how the model starts. When the total compute budget is fixed, none of the tested synthetic methods beats training on natural language alone.
Spreading synthetic examples through training prevents loss on those examples, but a general efficiency gain remains unproven.
The research
When unique natural-language text runs short, training compute can buy more passes over existing text or a phase on synthetic examples. Do those examples teach patterns that transfer well enough to justify diverting a fixed budget away from natural-language training?
The authors compare synthetic training with natural-language-only training and find no fixed-budget advantage for the tested synthetic methods.
The authors compare several synthetic-data curricula with natural-language-only training to ask whether a synthetic phase is worth its compute.
They test both additive compute and fixed-budget settings, measuring prediction on held-out FineWeb with bits per byte, where lower values mean fewer bits are needed to encode each byte.
In fixed-budget comparisons, no tested synthetic method beats natural-language-only training on that measure; repeating the same limited natural-language corpus performs better than every synthetic curriculum in the data-constrained setting.
The evidence answers the question for these methods and procedures, while leaving other curricula and schedules open.
These findings cover the tested synthetic curricula, training budgets, and language-modeling procedures. The interleaving runs were not hyperparameter-tuned, so whether a better schedule would improve natural-language performance or use compute more efficiently remains open.
What the evidence shows
- The compute allocation question
The paper studies whether language models benefit from a synthetic-data phase before natural-language training, especially when unique natural-language text is scarce. That phase, called pre-pretraining (PPT), is meant to teach patterns that carry over.
The practical research question is whether those patterns improve language modeling enough to justify the training compute, including compute that could instead be spent making more passes over natural-language text.
- A gain sensitive to initialization
First, the authors reproduce an earlier NCA result in an additive-compute setup: the synthetic phase is added before natural-language training rather than taking from its budget. The reproduced model has lower validation perplexity than its no-PPT comparison.
But changing the embedding initialization improves the no-PPT baseline enough to erase the distinguishable gap. The result shows both that the gain can appear and that it depends on the training setup.
- The fixed-budget comparison
The decisive test holds total compute fixed and compares synthetic training with natural-language-only training across three regimes. The authors measure prediction on held-out FineWeb using bits per byte, where lower means fewer bits are needed on average to encode each byte.
None of the tested synthetic methods beats the baseline on that measure. When unique natural-language data is limited to 100 million tokens, repeating that text performs better than every tested synthetic curriculum.
- A broader but mixed picture
The authors check whether the validation pattern also appears on tests of broader language abilities. DCLM CORE aggregates 22 tasks; BLiMP tests grammatical preferences by comparing the probabilities assigned to grammatical and ungrammatical sentences.
Their aggregate scores mostly match the natural-language baseline, with a few gains at the lowest compute settings. A natural-language C4 control scores in a similar DCLM CORE range, and individual tasks can improve or decline.
- What happens after the switch
Finally, the authors track performance on synthetic data through the transition to natural-language training. Front-loaded runs lose performance on those synthetic distributions after the transition.
Runs that use synthetic examples in 5% of each batch preserve and continue improving that performance, though natural-language results do not improve significantly in every case. The evidence supports reconsidering a separate front-loaded phase, but does not show that interleaving beats natural-language training in compute efficiency.
Deep
- Original paper authors
- Alexander Stern, Tim Rocktäschel, Acyr Locatelli, Laura Ruis
- Submitted to arXiv
- License
- CC BY 4.0