CRISP coordinates pruning and scheduling for edge MoE
CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers · arXiv:2610.37442
In plain words
Some language models choose a small set of expert networks for each piece of text. On a small device, the model may need to fetch expert weights from memory, and that can take longer than the computation.
CRISP studies which expert pairs are used often and which expert matters more within each pair. It keeps more of the experts that matter most and schedules their loading alongside computation.
On Mixtral-8x7B with cache space for 10% of expert weights, it generated tokens faster than MoE-Offload in the reported comparison.
The research
When expert weights cannot all fit in a device’s GPU memory, fetching them can take longer than computing with them. It remains unclear whether a system can reduce that cost while keeping more model capacity for the expert combinations and individual experts that matter most.
CRISP assigns more capacity to frequently used, influential experts and schedules their loading alongside computation to reduce data movement during edge inference.
CRISP uses combination frequency and expert role to choose how much of each expert to run, then schedules wider experts so their computation can overlap with loading narrower ones.
On Mixtral-8x7B with cache space limited to 10% of total expert size, researchers measured 8.3 generated tokens per second for CRISP and 3.9 for MoE-Offload, which schedules I/O without differentiated pruning. The result supports higher throughput for CRISP in this setting.
The throughput comparison is for Mixtral-8x7B with cache space set to 10% of total expert size; the cited result does not identify which test platform produced it.
The evaluation covered two models, two platforms, and specified workloads, while CRISP’s profiling can need refreshing when the model or workload changes.
What the evidence shows
- The cost of expert access
CRISP is an edge-inference framework that co-designs expert pruning and runtime scheduling for mixture-of-experts models. These models route each token to a subset of experts, but input-dependent routing means all expert weights must remain accessible.
When they cannot fit in GPU memory, loading weights from host memory can dominate inference time. The research asks how to reduce that movement without treating every expert combination alike.
- Combinations are unevenly used
The researchers profiled Mixtral-8x7B using 2,048 sequences sampled from the C4 validation set. The most frequently used 25% of expert combinations handled over 60% of token activations, while the least used 29% handled under 10%.
This unevenness suggests that assigning the same amount of model capacity to every combination may spend weight access on combinations that serve relatively few tokens. Which combinations are frequent varies by model layer.
- Experts can matter unequally
Frequency alone does not capture each expert’s role. In a masking comparison, the researchers replaced each expert’s output with zeros in turn and measured the increase in perplexity.
Across combinations, masking the expert designated dominant increased perplexity by an average of 2.64, compared with 0.71 for the auxiliary expert. This average indicates unequal measured effects across combinations, rather than a fixed difference in every pair.
- Match capacity and scheduling
CRISP uses these patterns to assign execution widths: both experts run at full width in hot combinations, the auxiliary expert runs at half width in warm combinations, and it is skipped in cold combinations. Neurons are reordered by importance so narrower execution can use a prefix of one stored expert copy.
During inference, the scheduler starts wider experts first and overlaps their computation with loading narrower ones.
- A bounded throughput gain
At a cache budget equal to 10% of total expert size, Mixtral-8x7B generated 8.3 tokens per second with CRISP and 3.9 with MoE-Offload, which schedules I/O without differentiated pruning. This shows higher token generation throughput for CRISP in that tested setting.
It does not establish the same gain for other models or budgets, nor does this whole-system comparison isolate which design component accounts for the difference.
Deep
- Original paper authors
- Congwei Zhang, Youming Tao, Yifei Zou, Yu Yang, Ruirui Zhang
- Submitted to arXiv
- License
- CC BY 4.0