Expert routing stays stable as copy timing changes
Prefetch-Steered Routing: Deterministic Cache-Aware Expert Selection for Offloaded Mixture-of-Experts Decoding · arXiv:2610.34721
In plain words
Some large language models keep parts called experts in computer memory because the graphics card cannot hold them all. The model must choose experts as it writes each token, while copies of experts may still be on the way.
This paper introduces a routing method that bases those choices on a predictable record of available experts, not on which copies have arrived at that moment.
In tested configurations, this made text-prediction results stay the same across copy schedules and repeated runs; speed and task scores still varied by model and workload.
The research
When a model chooses experts while their weights are being copied to the GPU, should changing transfer timing change the model’s output?
The paper tests this in cache-aware routing, where experts already on the GPU receive a selection bonus and asynchronous prefetch can affect which experts count as resident.
PSR uses a deterministic logical cache and next-layer predictions to steer expert routing while copy timing affects latency alone.
The authors first tested whether asynchronous expert copies can change what a cache-aware model computes. In Cache-Prior configurations, routing from the physical cache made perplexity shift with the number of copies in flight and vary between runs.
A logical cache kept perplexity fixed across those same tested scheduler settings, supporting timing-independent evaluation in this setup.
PSR uses that logical cache and inserts a limited number of experts predicted by the next layer’s router, giving them a routing bonus while copies are scheduled separately. On OLMoE, PSR’s teacher-forced latency was lower than exact routing with LRU and Cache-Prior at both tested residency levels.
Its average paired task-score differences from Cache-Prior were small and uncertain, so these results support a speed and quality trade-off in those measurements, not universal equivalence or improvement.
The main latency measurements use teacher-forced text and exclude prompt processing, while free-generation results depend on the workload and can reverse. The evaluations cover three models on one GPU, and OLMoE’s memory pressure was emulated.
What the evidence shows
- The timing problem
Prefetch-steered routing (PSR) is a cache-aware method for choosing experts during batch-one decoding when expert weights are offloaded to host memory. Each token activates experts, and missing weights must be copied to the GPU.
Asynchronous copies can overlap with computation, but if routing reacts to which copies have arrived, transfer timing can change the model’s choices. The paper asks how to evaluate routing consistently when timing becomes part of the choice.
- A reproducible cache state
In Cache-Prior, routing gives resident experts a score bonus, while the engine can prefetch experts asynchronously. The authors varied the number of copies in flight in tested configurations and measured perplexity, which indicates how well the model predicts text on the tested windows.
With physical-cache routing, perplexity shifted across scheduler settings and repeated runs. With logical-cache routing, it remained fixed across those settings and runs, showing that the evaluation can avoid this timing dependence in the tested setup.
- What PSR changes
PSR is the proposed method: next-layer router predictions enter a deterministic logical cache under a fixed insertion budget, making those experts eligible for a routing bonus. The router still selects the experts, while the engine separately schedules their physical copies.
The authors compared it with exact routing using least-recently-used caching and with Cache-Prior. On OLMoE, PSR had lower teacher-forced latency than both at 50% and 25% expert residency; its average paired task-score difference from Cache-Prior was small, with uncertainty intervals that included zero.
- A second offloaded model
The task tests measured performance on questions and prompts involving knowledge, reasoning, math, instruction following, and code. On Moonlight, where experts genuinely had to be offloaded, PSR was faster than exact routing with least-recently-used caching; at roughly Cache-Prior’s latency, it scored higher on average across eight tasks, with corrected gains on three and no significant losses. This supports transfer to that tested model, while leaving other models and individual tasks open.
- Where the speed lead stops
The speed advantage depended on the balance between copying and computation. On Qwen3.6, where expert copies were relatively cheap, PSR and Cache-Prior had similar teacher-forced latency, although PSR had lower divergence from exact routing and gains on two knowledge or reasoning tasks.
Free-generation latency also varied across prompt sets: on OLMoE at 50% residency, PSR was slower on HumanEval even though it was faster on GSM8K and IFEval.
Deep
- Original paper authors
- Siming Huang
- Submitted to arXiv
- License
- arXiv non-exclusive