Pruning preserves production-task quality better than expert merging
Prune, Don't Merge: Expert Compression of Mixture-of-Experts LLMs on Multilingual Production Tasks · arXiv:2611.12240
In plain words
Mixture-of-experts models contain many specialist parts called experts, but use only some for each token. This paper compares two ways to remove experts: pruning deletes experts whose gate-weighted outputs have the lowest average size during calibration, while merging combines groups of experts.
On five production tasks, pruning scored significantly better in more paired comparisons than merging. The largest gap appeared in generated text, where format failures and untranslated English openings accounted for much of the measured damage.
The research
Product tasks depend on more than whether an answer sounds right: invalid structured output, broken email layouts, or an English phrase left in another language can lower task scores.
This comparison challenges the expectation that pruning and merging are interchangeable ways to shrink routed experts, especially when generated output must meet a format contract.
The study compares expert pruning with expert merging on five multilingual production tasks to test which better preserves task quality after compression.
The authors compressed two models with each method at two removal levels, then scored them on the same validation rows across five production tasks. The tasks used exact-match scores, a production judge, or an email rubric with hard format gates.
Pruning was significantly better in 16 of 20 paired comparisons, while merging was better in two and two showed no significant difference. The pattern supports pruning in this test set, with the clearest advantage on generated tasks.
Each configuration was compressed once with one calibration set, and the study covers two models of one size class and five private production tasks. It measures task quality, not serving memory, latency, or throughput.
What the evidence shows
- Two ways to remove experts
The paper evaluates REAP expert pruning and REAM expert merging on five production tasks. Both methods remove the same share of routed experts after calibration on production conversations, but REAP deletes the least salient experts and keeps the rest intact, while REAM replaces groups with weighted averages.
The comparison asks whether these different ways of reducing the expert set preserve production-task quality to the same degree.
- Pruning wins more comparisons
The decisive comparison scores both methods on the same validation rows for two models and five tasks, at two removal levels. REAP was significantly better in 16 of 20 comparisons; REAM was better in two, and two showed no significant difference.
These are Holm-corrected paired comparisons among eight compressed checkpoints, not independent replications, but the pattern favors pruning within this evaluation.
- Generation widens the gap
For Qwen3-30B’s two generative tasks at 25% expert removal, pruning retained a mean 93.6% of the base model’s scores, compared with 62.5% for merging. This scoped comparison shows the methods separated most on generated work; it does not describe every task or model.
- Failures cluster in output form
For significant email-template score losses, hard-gate failures accounted for 89.8–95.2% of the loss, while the rubric scores among answers that passed changed less. This comparison points to output requirements as the main source of those measured email losses under the study’s relaxed rubric.
- A bounded production result
The results support a preference for pruning over merging on these production tasks, but each configuration was compressed once with one calibration set. The study covers two models of one size class and private task data, and it measures task quality rather than serving memory, latency, or throughput.
The observed advantage therefore does not settle how the methods compare under other models, calibration choices, or serving conditions.
Deep
- Original paper authors
- Tomas Rasymas, Povilas Kvedaras
- Submitted to arXiv
- License
- CC BY 4.0