Why sparse MoE models gain at low batch sizes
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models · arXiv:2610.38981
In plain words
Some language models use only a small group of experts for each token. Speculative decoding asks a smaller draft system to suggest several tokens, then has the main model check them together.
Earlier estimates suggested this could work poorly at small batch sizes because checking might load many experts.
Across ten models, many very sparse models instead had their strongest speedup near batch size one. The researchers found that nearby tokens often reuse experts, and built a method to estimate that reuse.
Its predictions closely matched expert counts in the models measured, though the method leaves some routing patterns out.
The research
If checking several proposed tokens can require loading many experts, does speculative decoding lose its advantage for sparse language models at small batch sizes?
Earlier estimates predicted limited gains there, but newer models divide work among many smaller experts, making that prediction uncertain.
The study measures expert reuse and GPU execution, then introduces an IPF framework to predict routing and test verification-cost tradeoffs.
The authors measure speculative decoding on ten sparse MoE models, where a target checks several draft-proposed tokens together. Many sparser models peak near batch size one, contrary to the earlier prediction that low-batch verification would load enough extra expert weights to limit gains.
They find that tokens reuse experts and that GPU kernels do not translate weight share directly into runtime. An IPF routing model closely predicts four-position expert counts across eight measured models, supporting the view that overlap helps explain verification cost.
The routing predictor captures pairwise overlap but omits higher-order patterns and changes in expert-selection rates by position. The study also estimates possible MoE draft-layer costs synthetically; it does not train such drafters or measure whether they improve acceptance length.
What the evidence shows
- A low-batch question
Speculative decoding has a target model check several tokens proposed by a draft in one pass. For sparse mixture-of-experts (MoE) models, earlier cost estimates treated verification as loading the weights of every distinct expert used across those positions.
That view predicted speedup would peak at intermediate batch sizes, because verification would load many experts at low batch sizes. The study asks whether that prediction describes recent, more finely divided MoE models on modern GPUs.
- The curves differ
The paper’s empirical study measures speculative decoding across ten sparse MoE models. Many sparser models with smaller experts show non-concave speedup curves that peak at or near batch size one, unlike the predicted intermediate-batch peak.
Less sparse models with larger experts, including Mixtral-8x7B, show the predicted concave pattern. The researchers also find similar low-batch curve shapes for Qwen3-30B and Qwen3-235B across multiple datasets, so those results are not limited to the base dataset.
- Tokens reuse experts
One reason expert-weight counts can misstate verification cost is that adjacent tokens often use some of the same experts. The researchers compare measured overlap with an independence baseline that preserves each expert’s observed selection frequency but assumes token positions route independently.
Across eight models, mean overlap is 35.1% for adjacent tokens and 26.4% at a separation of three tokens, against a 10.5% mean independence baseline at separations one to three. In six high-sparsity models, this temporal overlap reduces the four-position unique-expert count by 21–34% versus that baseline.
- Hardware changes the cost
Overlap alone does not account for how GPU kernels turn weight reads into runtime. In the profiled Qwen3-30B configuration on H100, MoE weights make up 67% of model weights but MoE kernels use 33% of forward-pass GPU time.
The authors also report higher achieved memory bandwidth for MoE than attention kernels and more cache hits during verification than ordinary decoding. These measurements show why expert bytes alone are a poor runtime proxy in that configuration; the detailed cache mechanism differs on B200.
- A routing-based estimate
The authors introduce a maximum-entropy Iterative Proportional Fitting (IPF) framework to predict how many distinct experts appear across four token positions using selection rates and pairwise overlap. Its skewed version’s largest relative error is 3.7% across eight measured models.
In tests using synthetic routing with dummy weights and real inference kernels, per-layer IPF tracks real-model target-efficiency curves more closely than uniform-random routing. This supports the framework as an approximation under tested conditions, not a complete model of routing or proof of transfer to other architectures.
Deep
- Original paper authors
- Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli
- Submitted to arXiv
- License
- CC BY 4.0