One Transformer serves five compute budgets with shared weights

In plain words

The research

What the evidence shows

• One model, five budgets

  Accordion is a Transformer method that uses one set of weights to serve five inference budgets. The smallest budget follows the ordinary path; larger budgets repeat a prefix of selected middle layers.

  This addresses a tradeoff raised by jointly training paths of different depths: a model might gain a deeper option yet lose quality on its shallow path. The authors’ question is whether fixed, shared paths can preserve both options without a separate model for every budget.

• Choosing which layers repeat

  To choose which layers to repeat, the authors compared eight strategies in a 1.61-billion-parameter design study. Each configuration was trained once on 80 billion tokens, with the same expected training depth, and evaluated on the same held-out web text.

  Fixed-prefix selection in layers 7–18 had the lowest observed validation loss at each repeated-layer budget and the lowest mean across the five budgets. Each strategy was evaluated with its specified training and inference paths.

• Checking shared and shallow paths

  A separate comparison tested one shared model against specialists, each trained for a single budget. With 80 billion training tokens per model and the same inference graph at each compared budget, the shared model’s validation loss was within 0.0030 nats of the specialist at budgets 0, 6, and 12.

  An equal-token control also found lower mean validation loss with uniform budget sampling than with a control that reserved replicas for the unlooped path; that difference was specific to the replica-reservation comparison.

• More computation, higher scores

  On Accordion-Base, the authors changed the inference path while keeping weights, tasks, examples, and evaluation configurations fixed. The 37 tasks assess knowledge and question answering, common sense, reasoning, mathematics, and code.

  Their score averages tasks within each category and weights the five category averages equally. The score rose from 47.05 at the unlooped budget to 50.06 at the largest budget, with higher scores on 36 of 37 tasks.

• Combining three domain models

  The method also carried five inference budgets into post-training. The authors combined updates from instruction-following, mathematics, and code models, then trained one student on its own responses using token-level signals from the domain-matched model at the same budget.

  Across five budgets, the student exceeded each corresponding domain model on its equal-task domain average, and all three paired 95% intervals were above zero. The result is scoped to those domain averages and evaluations.