One language model gains accuracy by spending more inference loops
En palabras sencillas
Elastic Latent Recurrence (ELR) is a way to train one looped language model to work at several depths. At one loop, it runs each unique layer once, like a same-size ordinary model; at more loops, it reuses a shared middle block to spend more computation.
In matched tests, researchers averaged accuracy across ten tasks. The 4B version was only 0.52 points below its ordinary counterpart at one loop, and the same checkpoint gained 7.25 points by four loops.
La investigación
Can one checkpoint serve both a low-compute setting and a higher-quality setting without losing the fixed-depth model's starting accuracy? The paper addresses this question because a large one-loop deficit would make extra loops a costly recovery step rather than a useful way to buy more accuracy.
ELR trains one looped language model to match a fixed-depth baseline at one loop, then improve as extra loops spend more inference compute.
The authors train 4B and 30B-A3B ELR models alongside non-recurrent baselines with the same backbone configuration, data, token budget, and ten-task scoring protocol.
Each ELR checkpoint is then evaluated at loop counts from one through four, so one experiment tests both the fast endpoint and the benefit of extra recurrent passes.
At one loop, the 4B model averages 65.13 versus 65.65 for its baseline, while the 30B-A3B model averages 66.08 versus 65.04. The same checkpoints gain 7.25 and 4.75 points respectively by four loops.
Near-baseline one-loop scores plus higher multi-loop averages support the claimed elastic compute-quality tradeoff.
The evidence covers only loop counts used in training and the reported evaluation harnesses. Koi-4B has no matched non-recurrent baseline, so its index result does not establish a matched one-loop comparison.
Análisis
Detalles
- Licencia
- CC BY 4.0