Layer repair changes which transformer layers are best to remove
In plain words
The paper introduces methods for choosing which whole transformer layers to remove when shortening a language model. Its key idea is to check each removal choice after fitting and applying a repair that helps replace the removed layers’ effects.
Across four tested model families, the greedy method had lower text-prediction perplexity than an existing method that used the same repair. The result is about those models and tests, not every language model.
The research
When a language model is shortened by removing whole transformer layers, can a method choose layers well before knowing how the remaining model will be repaired? Existing scores assess removals before repair, even though repair can change how a removed run affects the model.
The paper introduces layer-selection methods that account for linear repair, showing that the best layers to remove can change once repair is applied.
To measure that gap, the authors evaluated every candidate subset within modest removal budgets, with and without least-squares repair.
The best layout changed: without repair, exact optima at four or more removed groups split into runs; with repair, they became one block or a block plus one short run.
Their greedy method selects after applying repair. On four tested model families, it had 20% to 66% lower WikiText-2 test perplexity than a block-selection method using the same repair. This supports making repair part of the choice.
These results cover one least-squares repair, English text and multiple-choice tasks, and exact comparisons at modest removal budgets. They do not establish how the methods perform on multilingual text or open-ended generation.
Analysis
Deep
- Original paper authors
- Jianchang Su, Dawei Xiang, Yifan Zhang, Wei Zhang
- Submitted to arXiv
- License
- CC BY 4.0