A formula predicts how much rewritten text to train on
Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints · arXiv:2610.31867
In plain words
Language models can be trained on natural text, on rewrites made by another model, or on a mixture. Rewrites take computing power to make, so researchers need to know when they are worth that cost.
This paper introduces a formula that estimates how much rewriting to use while counting both training and rewrite-making costs. In tests on held-out text, the formula’s choices were close to the best among the fractions tested, at budgets not used to fit it.
The research
When natural training text is limited, rewritten text may help, but generating it uses computing power. Can researchers choose how much to use while accounting for that cost, and can a model trained on rewrites outperform the model that made them?
The authors fit a scaling law that predicts a useful share of rewritten training text while charging the compute needed to generate it.
The authors test whether a formula can choose how much synthetic rewriting to use when both training and generating rewrites consume computing power. The formula predicts held-out text loss and selects a rewrite fraction under a budget that includes both costs.
They fit it on smaller-budget runs, then compared forecasts with runs excluded from fitting. The forecasts stayed close to observed Paloma text-prediction scores and to the best tested rewrite fractions in data-limited runs. This supports the law as a guide within those tested settings.
These results concern language models trained on FineWeb-Edu with four rewrite styles and a limited set of rewriters. The compute estimate assumes rewrites match source length, generation is compute-bound, and rewriter training costs are excluded.
What the evidence shows
- A costly training choice
The rewriting scaling law is a method for estimating how much generated text to include when training a language model. It addresses two linked questions: how to account for the cost of generating rewrites, and whether a model trained on them can exceed its rewriter.
The law predicts held-out text loss from model size, training data, rewrite share, and, when natural data is limited, how much that data is repeated.
- Choose under a full budget
The law’s recommendation is the rewrite fraction that minimizes predicted loss after the compute budget includes both target training and rewrite generation. The authors fit it in two different regimes: one with plentiful natural text and an external rewriter, and one with a fixed natural-text seed set and a separately trained rewriter for each target size. The second regime also accounts for repeated passes over natural text.
- Predictions at new budgets
To check forecasts, the authors fit the law on lower-budget runs and compared its predictions with runs excluded from the fit. For compute-limited runs, a fit through 3x 10²⁰ training FLOPs forecast four Paloma text-prediction scores at 10²² FLOPs with a mean error of −1.04%; measured scores were 0.85% to 1.40% below forecast.
In data-limited runs, the selected fraction was at most 1.14% above the best tested fraction at two held-out budgets and four target sizes.
- A ceiling with exceptions
The comparison with the external rewriter addresses whether the rewriter’s own score is a ceiling. On Paloma, a 21.9-billion-parameter target trained with a 32% rewrite fraction scored better than the 1.7-billion-parameter rewriter on 10 of 16 sources.
Its average across the sources was still slightly worse, 0.983 versus 0.972, so the result shows a source-by-source exception to a hard ceiling rather than an overall win.
- Gains depend on the task
Adding rewrites also helped the data-limited targets on Paloma: at each tested size, the law-selected positive fraction scored better than natural-only training at the same budget. Other tasks were less uniform.
For the 3-billion-parameter target, mixtures beat natural-only runs on math and code at every tested budget, while natural-only was best on question answering through W=6. The gains therefore depend on what is measured.
Deep
- Original paper authors
- Nicholas Roberts, Ahmed Ahmed, Suhas Kotha, Percy Liang, Yu Wang
- Submitted to arXiv
- License
- CC BY 4.0