Two fully trained loops can match large models with fewer parameters

When a model reuses the same transformer block, running it more times adds computation without adding parameters. Training through fewer repetitions saves compute, but may weaken the benefit of those extra passes. Under a fixed training budget, should compute go to more forward loops or to training each loop?

The authors fit a law that treats looping as added effective parameters, then use it to compare truncated and fully trained loops under fixed compute.

The authors ask how to divide a fixed training-compute budget between running a shared transformer block more times and training those loops with gradients. They fit a scaling law that estimates validation loss from model size, training tokens, and separate forward and backward loop counts.

The law predicts that truncation can save compute when forward depth is fixed. But matched-compute training runs found lower validation loss with four fully trained loops than with eight forward loops trained through two, at five sizes and one token budget. This supports choosing fewer fully trained loops in that tested comparison.

These results concern dense models and validation loss, not downstream task performance or reasoning ability. The experiments use at most 16 forward loops and keep the ratio of once-run to recurrent parameters fixed.