A revised test narrows the compute gap for diffusion models

A widely cited estimate says masked diffusion language models need about 16 times the training compute of autoregressive models to reach the same validation performance.

Does that gap still hold when diffusion models are scored by the likelihood of their usual token-reveal order and use block-based designs?

The authors compare language-model compute scaling with updated likelihood scoring and block-based designs to measure how much the earlier gap changes.

The authors compare masked diffusion with autoregressive language models using a revised compute-matched scaling analysis. For diffusion, DUEL scores exact negative log-likelihood under the model’s usual top-1 confidence order for revealing tokens.

The authors fit performance across training budgets and compare the compute each family needs to reach the same loss.

With the same full-sequence diffusion checkpoints, the fitted compute multiple at 10^20 FLOPs falls from 17.8 with the evidence lower bound to 12.8 with DUEL. Encoder–decoder block diffusion narrows it further, reaching 3.0 at block size 4 in the fitted comparison. A larger-budget check at block sizes 16 and 32 finds gaps close to the fitted predictions.

The comparison covers the models, data, training recipe, scoring rule and fitted compute range tested here. The study used one random seed and one learning-rate and batch-size configuration, and DUEL was evaluated only with the top-1 confidence token-reveal rule.