In plain words
When a request arrives, a server must decide what to work on. For block-diffusion language models, a long answer does not necessarily take more work: each output block can need a different number of steps, and the server cannot know that number at the start.
DWISE estimates the work left in each request, updates its estimate after the first output step, and shares each round between prompt and output blocks. Across five tested models and two arrival patterns, its mean request latency was lower than SGLang’s, with a 5.4x geometric mean speedup across the ten settings. That average does not promise the same gain for every model.
The research
When a server handles block-diffusion language models, output length does not reliably reveal how much work a request needs. And prioritizing new prompts can keep delaying requests already producing output. Can the server estimate remaining work well enough to choose which requests to run each round?
DWISE estimates each request’s remaining prompt and denoising work, then schedules both kinds of blocks together to reduce serving delays.
DWISE is a runtime scheduler for block-diffusion language models. Because output length does not reliably show how many denoising rounds remain, can a server estimate work well enough to decide which requests to run first?
DWISE combines estimated remaining denoising rounds with known unprocessed prompt blocks, then schedules both kinds of blocks in shared rounds. Across five models and two arrival patterns, its mean request latency had a 5.4x geometric mean speedup over SGLang across ten settings. Accuracy on the tested single-answer and long-context tasks stayed within one percentage point of SGLang’s.
The main comparison covers five models with a 256-token output budget on one GH200 serving instance per model. Broader output-budget and multi-GPU tests cover LLaDA2.1-mini only, and DiffusionGemma cannot use mixed-phase batching.
What the evidence shows
- The scheduling mismatch
DWISE is a runtime scheduler for block-diffusion language models. It addresses a mismatch between familiar serving rules and how these models work: output length does not reliably indicate the number of denoising rounds, and processing a prompt does not immediately produce the first output token. The scheduling question is which prompt or output blocks should run in each round.
- Estimating work left
DWISE estimates denoising work from prompt features when a request arrives, then refines that estimate once using hidden states from the first denoising round. It adds the estimated remaining rounds to the exact number of unprocessed prompt blocks.
This gives it one work estimate for ranking requests across both phases, although the denoising portion remains an estimate.
- Latency across settings
The main evaluation compared DWISE with SGLang across five block-diffusion models and burst and burst-plus-Poisson arrivals. Across the ten model-by-arrival-pattern settings, DWISE’s mean end-to-end request latency had a 5.4x geometric mean speedup over SGLang. The result is an aggregate across settings, with each setting’s mean averaged over three runs.
- What the comparisons show
A control on LLaDA2.1-mini kept DWISE’s mixed-phase batch former fixed while changing the ranking method.
DWISE’s ranking had lower mean latency than the best baseline in each arrival pattern, supporting a separate contribution from how requests are ordered. On the reported single-answer and long-context tasks, DWISE’s accuracy stayed within one percentage point of SGLang’s.
- Where the result stops
The gains depend on the model and serving conditions. DiffusionGemma cannot share a round between prompt encoding and denoising, so DWISE only reorders its requests; its mean-latency speedup over SGLang is 1.1x, and p99 latency rises under burst-plus-Poisson arrivals.
The main results use a 256-token output budget and one GH200 instance per model; broader hardware and budget extensions cover LLaDA2.1-mini only.
Deep
- Original paper authors
- Lie Li, Wen Li, Junxiao Shen, Jindong Gu, Guosheng Hu
- Submitted to arXiv
- License
- CC BY 4.0