EN
Frozen MoE decoding can select channels without calibration
Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data · arXiv:2610.37386
In plain words
Mixture-of-experts models send each token through selected expert networks. During generation, reading all the selected experts’ weights can take time, so channel selectors compute only some channels.
This paper introduces a reader that uses the model’s weights to choose channels and estimate what the skipped ones would contribute.
Across three models and three tests, the reader kept a mean 98.1% of full-model accuracy across nine model-test pairs while reading 17.8% of the full model’s routed-expert weights. That mean summarizes those pairs; it does not describe every result.
The research
A calibrated cutoff is meant to give channel selectors a known read budget and improve accuracy at that budget. But during generation, the number of channels above a fixed cutoff can change with the input. Does calibration deliver either promised benefit when frozen MoE models are evaluated?
The authors compare fixed score cutoffs with fixed channel counts, then introduce a weights-only reader that estimates skipped channels.
To test calibration’s claimed value, the authors compared each selector’s cutoff with keeping the same score’s highest-ranked channels at the cutoff’s target read budget. Top-k tied or scored higher in 40 of 43 paired comparisons, including all 23 generation-task pairs; one Gemma 4 generation case showed a severe threshold failure. These results show no consistent accuracy advantage for calibration in the tested comparisons.
The threshold comparisons concern channels in frozen MoE models at the tested budgets. The reader was evaluated on three post-trained models up to 35 billion parameters, and its kernels were tuned for one GPU type and one read budget.
What the evidence shows
- What calibration should provide
Mixture-of-experts models route each token through selected expert networks. Channel selectors reduce the weights they read by computing only some channels in those experts.
A common approach sets a score cutoff using sample data, aiming to meet a chosen read budget. The paper asks whether that cutoff actually delivers the budget and whether it improves accuracy over keeping a fixed number of high-scoring channels.
- The read budget drifts
A fixed score cutoff can admit different numbers of channels for different inputs and tokens. The authors checked actual routed-expert reads during decoding, measuring them as a share of the full model’s routed-expert reads.
Across 72 runs of Prox and the authors’ reader with calibrated thresholds, reads fell below the calibration target in every run. For Prox calibrated to 18%, the mean was 12.9% across nine model-test pairs.
- Testing accuracy fairly
To isolate the keep rule, the authors ran each selector with the same score under its calibrated cutoff and under top-k selection, which keeps a fixed number of the highest-scoring channels. They compared accuracy at the cutoff’s target budget.
Top-k tied with or beat the cutoff in 40 of 43 paired comparisons, including all 23 generation-task comparisons; a tie means no difference was detected beyond two paired standard errors.
- A failure and a new reader
The aggregate comparison has a consequential case: on one Gemma 4 generation test, Prox with a calibrated channel cutoff fell into repetition loops, while replacing that cutoff with top-k at nearly the same read share sharply raised accuracy and reduced answers reaching the token limit.
Separately, the paper introduces a reader built from model weights; across three models and three tests, it retained a mean 98.1% of full-model accuracy at 17.8% of routed-expert reads.
Deep
- Original paper authors
- Geunhyoung Kim, Sung Ju Hwang
- Submitted to arXiv
- License
- CC BY 4.0