A coordinated system makes low-bit language models generate in blocks
In plain words
BitMTP is a framework that helps very low-bit language models write several tokens in one step. It trains one model both to write one token at a time and to draft short blocks, then uses different ways to decide which drafts to keep. Its runtime also tracks which accepted tokens still need to be added to the model’s history.
On an NVIDIA H20, the tested models generated faster on average with BitMTP’s hybrid method than with their supervised autoregressive versions, while their average scores across five generation tests stayed close. The result applies to those models and tests.
The research
Can drafting several tokens at once actually speed up generation when a model uses very low-bit computation? Extra work to form drafts and update the model’s history can erase the savings, and a naive combination lost speed and task quality in one low-bit test. BitMTP asks whether coordinating training, decoding, and runtime can address that tension.
BitMTP jointly trains block drafting and next-token generation, then aligns decoding with a runtime that reuses low-bit computations and model history.
BitMTP trains one model for both next-token generation and multi-token drafting, then offers decoding choices for accepting drafts. Its runtime keeps accepted tokens aligned with the model’s causal history so it can reuse low-bit computations as generation proceeds.
On an NVIDIA H20, the authors compared hybrid BitMTP with supervised autoregressive routes across five generation tests. Mean output-token speed was higher for BitNet and two Bonsai sizes, while their mean task scores stayed close.
The tests cover instruction following, reasoning, and code, so these averages support a bounded speed improvement across the tested tasks.
These results cover the named 1.7B–4B models, five generation tests, and CUDA-based GPU setups.
The paper does not specify whether each BitMTP model and its supervised autoregressive baseline started from the same checkpoint; larger models, longer contexts, and direct use on non-CUDA accelerators remain unvalidated or unsupported.