A compact state speeds up hybrid model serving
SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving · arXiv:2611.14246
In plain words
Some language models keep a changing record of earlier text as they generate each new token. Systems often store that record as a large, dense object and repeatedly read and rewrite it.
SinkState stores it using a smaller set of directions, updates the important directions as new text arrives, and occasionally tidies the rest on the CPU.
In tests of four hybrid models on one GPU, SinkState used less recurrent-state memory and decoded faster on average than dense-state systems. Accuracy changes across the tested model-and-task pairs had no consistent direction.
The research
Repeatedly reading and rewriting a model's full history state can consume memory and slow generation. SinkState's results suggest that, for the four tested hybrid models, that cost can be reduced while measured task accuracy has no consistent shift.
This changes the expectation that recurrent states must stay dense during serving, within the tested models and setup.
SinkState adaptively stores and updates recurrent states in a compact form, then compares serving speed, state memory, and task accuracy with dense-state systems.
The evaluation compared SinkState with dense-state implementations from Mamba-SSM and FLA on four hybrid models. Both systems used the same batch size within each task test, with greedy generation at the test's default output length.
SinkState averaged 1.25 times the baseline decoding speed and 1.9 times smaller peak recurrent-state memory. Across 24 model-and-test pairs, accuracy changes had no consistent direction and averaged +0.3 points, supporting a bounded finding of lower state memory and faster serving without a consistent accuracy shift in these tests.
The serving results cover four Mamba-2 or GDN models on one GPU system. The reported averages summarize those tests, and other implementation differences between SinkState and the dense-state baselines are not established as controlled here.
What the evidence shows
- The cost of dense states
SinkState is a serving system for hybrid language models that keeps recurrent states in a changing low-rank form during inference. These models use linear attention to summarize earlier text in a state, but dense-state systems read and rewrite that entire state at each decoding step.
The paper asks whether storing fewer important directions can reduce that repeated work while retaining task accuracy.
- Evidence for concentration
The profiles show that query, key, and value matrices concentrate much of their energy in a few leading directions. This supports the design premise for compact state storage, but does not itself establish a serving improvement.
- What the system comparison shows
The decisive system comparison was with dense-state implementations of Mamba-SSM and FLA. Across four evaluated hybrid models and six end-to-end task tests, SinkState's average decoding speed was 1.25 times the baseline, while average peak recurrent-state memory was 1.9 times smaller.
The measures describe serving time and recurrent-state memory in those tests, not a universal improvement for every model or workload.
- Accuracy has no clear shift
Across the tested model-and-task pairs, accuracy changes went in both directions and had a small positive average. This does not establish a consistent accuracy gain or loss, and individual pairs still included decreases.
- The role of consolidation
The control suggests CPU consolidation helps limit recurrent-state memory growth, with a larger contribution in the tested Qwen models than in the Nemotron models. Since the comparison's workload and other fixed conditions are unspecified, it does not establish the effect across serving settings.
Deep
- Original paper authors
- Haolin Jiang, Weihang Li, Zhongzhu Zhou, Jinwen Pan, Santosh Pandey
- Submitted to arXiv
- License
- arXiv non-exclusive