A new drafter speeds up verified language-model generation
ChainFlow: Few-Step Latent Flow Matching for Speculative Decoding · arXiv:2611.08669
In plain words
A language model can draft several possible next words and check them together, but drafting them all at once can miss how one word affects the next. ChainFlow is a drafter that uses the model’s recent internal representations to sketch future ones, then uses proposed words to guide later guesses. The target model checks the draft and chooses what to keep.
On one Qwen3.5 model and eight English test areas, ChainFlow’s branching draft produced more tokens per second than two comparison drafters in the reported single-request setup. The finding applies to that measured setup; at higher request loads, the tree can be slower than ordinary generation.
The research
Generating text requires repeated work by a large language model, so the cost of proposing each next token matters.
This result changes the comparison for one tested setup: a drafter can use the model’s own recent internal states and token feedback to outperform two alternatives in serving speed, while the advantage depends on load and decoding conditions.
ChainFlow drafts tokens by moving compressed representations of a target model’s recent internal states toward future states, then uses token feedback before the target checks them.
In speculative decoding, a target model checks a drafter’s proposed tokens. ChainFlow uses compressed representations of the target’s recent internal states to propose a chain or tree, then the target verifies those proposals.
On Qwen3.5-4B, its tree produced output faster than EAGLE-3 and DFlash across eight English test domains.
The comparison used matched drafter sizes and the same training data, with greedy decoding and one request at a time. It supports a speed advantage for this configuration in that tested setting.
The English comparison covers one Qwen3.5 model size and eight test domains, and the design controls use a smaller model on GSM8K. Tree serving needs a patched engine and greedy decoding; 4–13% of prompts differed from plain greedy outputs in the reported runs.
What the evidence shows
- A drafter for future tokens
ChainFlow is a speculative-decoding drafter: it proposes tokens for a target language model to check. The design tackles a tension in drafting: generating proposals one by one preserves dependencies but costs more as the draft grows; proposing them in parallel saves that sequential work but can miss how earlier proposals shape later ones.
ChainFlow uses recent target-model representations to propose future tokens, then adds token feedback to recover some of those dependencies.
- Feedback helps in a control
The comparison suggests token feedback can improve drafting in this design, but it does not show feedback alone caused the change: the table does not establish that other training conditions matched.
This control used a Qwen3.5-0.8B target on GSM8K, so it does not establish the same effect in the main comparison.
- The English comparison
The central comparison measures output tokens per second relative to ordinary autoregressive decoding. On Qwen3.5-4B, pooled across eight English test domains, ChainFlow’s tree reached 1.11x the speed of DFlash and 1.39x that of EAGLE-3.
The drafters had matched sizes and the same training data, supporting an advantage for this tested tree configuration over those two alternatives in this setting.
- Adaptation beyond English
The evidence extends to sequential adaptation for Turkish, German and French: ChainFlow gained more speedup over the two comparison methods in each added language. English performance declined slightly for all three methods.
This points to a promising result across these added languages, while leaving the size of the gains and their transfer beyond the tested languages open.
- The boundary of the result
The speed advantage depends on serving load and output conditions. In the Qwen3.5-4B concurrency test, the tree was faster only up to four concurrent requests; at eight, it reached 0.94x the output-token speed of autoregressive decoding.
Tree verification also required a patched engine and greedy decoding, and some prompts produced outputs different from plain greedy decoding.
Deep
- Original paper authors
- Selim Aktas, Yusuf Erdem Erdem, Ahmed Zeer, Himmet Toprak Kesgin, Muzaffer Kaan Yuce
- Submitted to arXiv
- License
- Not specified by source