A measured table helps vLLM adjust draft length
TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding · arXiv:2610.38968
In plain words
A model can write faster by letting a smaller drafter guess upcoming tokens for it to check. But guessing more can waste work when many requests are being handled at once, so one fixed number of guesses may not suit every load.
TableTune measures serving time and how often guesses are accepted, then makes a vLLM table that changes the number of guesses with load. In one H100 test with Qwen3-4B and EAGLE-3 on code prompts, its table stayed within 4% of the fastest fixed choice at each tested load.
The research
When requests arrive at different loads, how can vLLM choose how many tokens to guess without relying on an operator to find the right fixed setting? The paper tests whether measuring serving costs and guess acceptance can produce a table that adapts to that changing load.
TableTune measures serving costs and guess acceptance, then builds and updates a vLLM table that selects draft length by request load.
On an H100, with Qwen3-4B, EAGLE-3 and code prompts, TableTune’s table lost at most 4% of output-token throughput against the fastest fixed length at each load from 1 to 128 requests. That reference is selected after testing each fixed length, so it is not a setting known in advance.
This result suggests measured load-specific choices can approach the per-load best in this setup, while a single length is less uniformly suited as load changes.
The tuner measures serving-step time and accepted guesses; its controller refreshes acceptance estimates from live requests and rebuilds the table.
These results cover specific models, GPUs, serving engines and workloads. The paper does not evaluate longer prompts beyond its tested range, mixed or recorded traffic, larger mixture-of-experts models, multi-node serving or disaggregated serving.
What the evidence shows
- A choice that changes with load
TableTune is a method for vLLM that measures and updates the table used to choose how many tokens to guess during speculative decoding. A smaller drafter proposes tokens, and the target model checks them.
The choice matters because extra guesses add work, and the best number can change as the number of requests handled together changes. The paper asks whether a measured table can track that changing load more effectively than relying on a fixed choice.
- Fixed lengths trade places
In the paper’s H100 example, Qwen3-4B with EAGLE-3 handled code prompts. Length 3 had the highest output-token throughput at low load, while length 1 was best at 128 requests.
Each fixed choice lost throughput where the other was better: length 1 lost 15% at one request, and length 3 lost 12% at 128. Step time also depends on hardware, engine and batch size, but is stable enough to measure offline; acceptance depends on traffic and should be tracked online.
That distinction motivates TableTune’s tuner, which measures step times, and its controller, which updates acceptance estimates from live traffic. The comparison shows why a single setting may not fit the full tested load range.
- Measure, then update
To create its table, TableTune measures serving-step time for different batch sizes and draft lengths on the configured vLLM setup. It estimates how often guesses are accepted using sample prompts, then combines those estimates with timing to select a draft length for each batch size.
The result is a table written in vLLM’s format. During serving, a plugin uses live acceptance counts to refresh its estimates and rebuild the table.
- Close to a per-load reference
On the H100 code-prompt test, TableTune’s table had at most 4.0% less output-token throughput than the fastest fixed length at each tested load, from 1 to 128 requests.
That reference is selected after benchmarking each fixed length, so it is an informative upper comparison rather than a ready-made operator choice. The authors’ adaptive-controller reimplementations had larger worst losses in this setup, though their available draft lengths differed from TableTune’s.
- Repair and deployment results
A separate experiment started from a documented vLLM table whose batch ranges had been divided by four, with an acceptance estimate from another workload. Against that same table left unchanged, the controller raised throughput by as much as 58% at a tested load.
In an llm-d deployment with two Qwen3-32B replicas, the tuned table also outperformed fixed length 3 and vLLM’s documented table at 256 to 1,024 open requests. These results support the method in the tested conditions, not every serving setup.
Deep
- Original paper authors
- Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard
- Submitted to arXiv
- License
- CC BY 4.0