LayerTrim compresses reasoning models while preserving their depth
In plain words
Making a reasoning model smaller can hurt its ability to solve problems that need long chains of thought. This paper introduces LayerTrim, a method that uses examples of the model’s reasoning to choose which internal features to remove. It keeps the attention parts and the model’s depth, and does not train the model again.
Across three models, LayerTrim had the highest average accuracy among the pruning methods tested at both size cuts. That finding applies to the five math, science, and coding tests in the study.
The research
When a reasoning model is compressed, can it keep answering tasks that depend on long chains of thought? Existing pruning methods built for non-reasoning models can sharply damage this performance, leaving a research question: which parts of a model can be reduced while preserving those abilities?
LayerTrim measures redundancy and feature contribution on a model’s reasoning traces, then trims selected MLPs while keeping attention and depth.
The researchers compared LayerTrim with six pruning methods on three reasoning models, using five tests for math, science, and code. Each test score measures accuracy on its task; LayerTrim had the highest average among the compared methods for all three models at both tested size cuts.
On Qwen3-14B at the larger cut, its average across five tests was 65.1%, versus 58.3% for RESP. This supports better measured accuracy in these comparisons.
The evaluation covers three reasoning models, five tests, and fixed pruning budgets and layer-allocation settings. It does not establish that the same results hold for other architectures, model sizes, or reasoning tasks.
Analysis
Deep
- Original paper authors
- Woosang Lim, Haneul Yoo, Vincent-Daniel Yun, Sungjoo Yoo
- Submitted to arXiv
- License
- CC BY 4.0