A serving system cuts draft work that often goes unverified
ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding · arXiv:2611.13442
In plain words
A language model can draft several possible next words together, then check those drafts before using them. At high load, one system drafted 15 tokens per request but checked an average of only 3.77, wasting much of the drafting work.
ForeSpark estimates which draft lengths are promising, chooses how much to draft and check for each request, then spends the checking budget on the strongest candidates. On one Qwen3-14B chat workload, it produced more output tokens per second than DSpark at the same user interactivity.
The research
When serving language models, a fixed-size draft can consume computation on tokens the system will never check. Choosing how much to draft for each request changes the decision about where serving time goes, and the reported results show higher throughput and request capacity in specified workloads.
ForeSpark implements work-conserving speculation, jointly choosing each request’s draft length and the batch’s verification budget to reduce wasted serving computation.
ForeSpark estimates likely acceptance before drafting, chooses request-specific draft lengths and a verification budget, then uses scores from the realized drafts to decide what to check. This differs from drafting a full block for every request and trimming only at verification.
In tests, ForeSpark improved throughput at matched user interactivity and handled a higher request rate at a specified latency target. At the largest batch size in the Qwen3-14B decode-only test, ForeSpark reduced drafting time per output token by 62% relative to ForeSpark w/o JS, which shares its checkpoint but drafts full blocks with adaptive verification. This control supports the view that coordinated allocation reduces wasted work.
The largest reported gains come from specific Qwen3-14B workloads, and the main comparison changes both scheduling and drafter backbone. Global sorting can also leave an individual request without a draft in a step, potentially skewing per-request speedups.
What the evidence shows
- The cost of full drafts
At high load, drafting a full block for every request spends computation on tokens that verification may discard. WCS addresses this by choosing each request’s draft length before drafting, so the system can avoid that cost.
- Joint choices improve throughput
ForeSpark applies WCS by estimating how many tokens are likely to survive verification at each possible draft length, then scheduling a length for every request alongside a batch verification budget. On Qwen3-14B’s decode-only ShareGPT workload, it produced up to 1.77 times DSpark’s output tokens per second at matched user interactivity.
The reported ratio includes only operating points where both systems reached at least half their peak throughput. This comparison is bounded to that workload and also changes the drafter backbone, so it does not isolate scheduling alone.
- Capacity under latency targets
Capacity measures how much incoming traffic a system can handle while 90% of requests meet both latency targets.
ForeSpark sustained a higher request rate than DSpark in this Qwen3-14B end-to-end test, supporting a capacity gain under those conditions; the comparison also changes the drafter backbone.
- Checking the source of gains
SPEED-Bench uses qualitative prompts to compare how many draft tokens are accepted per verification step, plus one. With every drafted token checked and adaptive verification off, ForeSpark’s average on Qwen3-14B was close to DSpark’s; ForeSpark and DFlash2 had identical acceptance because they share a drafter checkpoint.
This diagnostic supports the authors’ view that serving gains reflect allocation rather than higher acceptance, but it does not itself measure serving performance.
- Less work, unevenly allocated
A shared-checkpoint control provides another check: at the largest batch size in the Qwen3-14B decode-only test, joint scheduling reduced drafting time per output token by 62% compared with full-block drafting plus adaptive verification.
The result links shorter scheduled drafts to less drafting time in that condition. It does not mean every request benefits equally; global sorting can leave some requests undrafted in a step.
Deep
- Original paper authors
- Jaeyong Song, Daniel Wang, Pratyush Patel, Jonny Kong, Preyas Shah
- Submitted to arXiv
- License
- arXiv non-exclusive