KVStream uses idle storage to speed long-context model serving
KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving · arXiv:2610.31827
In plain words
Long prompts can reuse saved pieces of earlier text, but those saved pieces may not fit in a computer’s fastest memory. Reading them from storage can slow down the model, even when another storage system is sitting mostly unused.
KVStream is a storage engine that reads pieces from multiple storage tiers at once. In one offline test using a 110-billion-parameter model on a single testbed node, it processed the long-document question-answering dataset 1.49 times as fast as LMCache. Results depend on the workload and setup.
The research
When reused prefixes no longer fit in GPU and host memory, storage reads can delay inference while a mounted parallel file system carries little retrieval traffic. Can a serving system use that extra storage capacity when read speeds vary from step to step?
KVStream pools cached-prefix reads across storage tiers and lets each tier claim more work as it finishes, using their combined read capacity during inference.
In long-context model serving, saved prefixes can exceed GPU and host memory, leaving storage reads on the inference path. It remains unclear whether a mounted parallel file system can help alongside local storage when existing retrieval reads requests sequentially.
KVStream pools a step’s reads and lets each storage tier take more chunks as it finishes.
In separate tests, researchers found higher offline throughput on a long-document workload and lower online time to first token under bursty arrivals, consistent with faster retrieval helping serving in those settings.
The gains depend on storage reads remaining important and prefix reuse being substantial. With reuse synthetically reduced to 4% in the coding-agent trace, KVStream’s mean time to first token was slower than LMCache’s; replicated writes also consume bandwidth.
What the evidence shows
- A storage bottleneck
KVStream is a storage engine that retrieves saved key-value cache prefixes across multiple storage tiers during large language model inference. Its target is long-context serving where reused prefixes spill beyond GPU and host memory.
The motivating question is whether the parallel file system already mounted by a server can help retrieve those prefixes instead of leaving its bandwidth mostly idle.
- Why retrieval matters
Researchers measured LMCache serving LooGLE, a long-document question-answering dataset, on Testbed-1. Across four model sizes, prefix retrieval took 80–88% of processing time, and reads proceeded request by request.
This diagnosis makes faster retrieval relevant to inference performance in that setup, while the share itself is specific to that workload and testbed.
- Sharing reads as tiers finish
KVStream pools the storage reads for a whole inference step, delays writes behind the forward pass, and copies each new cache chunk to every tier. As a tier finishes a batch, it claims more unread chunks.
This lets the tiers share work according to when they complete rather than according to a fixed split chosen in advance.
- Two serving results
In offline LooGLE serving on one Testbed-1 node, the 110B model processed tokens 1.49 times as fast with KVStream as with LMCache. In a separate online evaluation, mean time to first token for 4,000 coding-agent requests was 6.07 times faster at arrival variation coefficient 4. These are different measures under different settings, rather than one general performance figure.
- Where the gain narrows
An incremental offline ablation found little throughput change from batching or delayed writes alone; adding a second tier and then completion-driven routing raised throughput in that setup. But when prefix reuse was synthetically reduced to 4% in the online trace, KVStream was slower than LMCache on mean time to first token.
Replicated writes cost bandwidth, so the result depends on storage demand being large enough to offset that cost.
Deep
- Original paper authors
- Avinash Maurya, Jaiaid Mobin, Jie Ye, M. Mustafa Rafique, Franck Cappello
- Submitted to arXiv
- License
- CC BY 4.0