ReplaySSM cuts recurrent-state traffic during language-model decoding

When a model's state-space layers write a large recurrent state after every token, can those writes be reduced without changing the outputs? The question matters because these state updates can be a major part of decoding latency, even in hybrid models whose attention cost grows with context. The paper tests a different way to represent and update that state.

ReplaySSM stores recent inputs beside a checkpoint, computing outputs between periodic state updates to reduce memory traffic and support rollback.

The paper asks whether state-space layers must write their full running state after every token, even when each update does little work. ReplaySSM stores recent inputs beside a checkpoint and updates it periodically, aiming to reduce those writes while retaining the same outputs.

In four hybrid models at batch size 256, its whole-model decode step was 1.20–1.47 times faster than tuned conventional vLLM decoding. This supports the value of changing state management in those tests; separate experiments also show why keeping low-precision reads out of recurrent updates matters.

These results come from four hybrid models tested on one GPU each without tensor parallelism; the low-precision experiments used emulated quantization. The measurements do not establish the same performance or accuracy elsewhere.