Separating expert work helps some rack-scale language models
FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink? · arXiv:2611.07149
In plain words
Some language models use specialist parts called experts alongside attention, which tracks what matters in the prompt. FastAFD is a serving engine that runs attention on one group of GPUs and expert calculations on another, overlapping the two jobs.
On one GB200 NVL72 rack, it was compared with a tuned setup that kept the jobs together but spread experts across the rack.
Counting every allocated GPU, including FastAFD’s expert GPUs, standard-decoding throughput ranged from a slight shortfall (Qwen3.5 at 16K) to a small advantage over that setup.
The research
Choosing whether to separate attention from expert calculations matters because the dedicated expert GPUs consume part of a rack’s capacity.
The measured comparison changes the expectation from “separation is faster” to a workload-specific choice: some models gained, while others matched or slightly trailed tuned colocated serving.
FastAFD separates attention from expert work and tests whether that design beats tuned, rack-wide colocated serving on GB200 NVL72.
The authors measured decoding on one GB200 NVL72 rack, comparing FastAFD with vLLM wide expert parallelism tuned for each model and prompt length. Counting every allocated GPU, including FastAFD’s dedicated expert GPUs, the standard-decoding ratios ranged from 0.99x to 1.17x.
That range supports a conditional answer: separation can help, but the result depends on the model and prompt context. In a second test of two models, speculative decoding produced ratios above parity across the six tested workloads.
The hardware measurements cover one GB200 NVL72 rack, three models in standard decoding, and two of them in the reported speculative comparison. The proposed gains from placing expert calculations on other accelerators are modeled, not measured on those accelerators.
What the evidence shows
- A stronger rack-scale comparison
FastAFD is an inference engine for mixture-of-experts language models that separates attention from expert calculations, then overlaps the two jobs across alternating microbatches.
The paper asks whether this attention–FFN disaggregation (AFD) design can improve decoding on a GB200 NVL72 rack against a strong alternative: vLLM wide expert parallelism, which keeps the jobs together while spreading experts across the rack.
- The rack narrows the gap
Counting throughput per allocated GPU, including FastAFD’s dedicated expert GPUs, against tuned vLLM wide expert parallelism tests whether separation pays after its rack-capacity cost.
Across the tested models and prompt contexts, results ranged from near parity to gains, with one slight shortfall; the comparison supports a workload-dependent answer, not a consistent win.
- Speculation shifts the balance
The comparison indicates that speculative decoding shifts the tradeoff in FastAFD’s favor: in the two tested models and prompt contexts, its extra expert work could be hidden behind attention enough to offset the cost of dedicating GPUs to experts. This result is limited to the tested speculative setup.
- A model explains the tradeoff
The authors explain the workload differences as a balance: separating experts can make larger attention batches possible and hide expert work, but dedicated expert GPUs consume capacity without producing tokens.
Their model predicted measured throughput with 2.9% mean error across 65 deployments at or below each workload’s measured best ratio, supporting estimates from colocated profiles within that tested range.
Deep
- Original paper authors
- Yichao Fu, Yuxuan Zhang, Minshen Zhang, Ruitian Wang, Junda Chen
- Submitted to arXiv
- License
- arXiv non-exclusive