What to Read, Keep, and Write During Model Inference
A model serving long documents may spend work finding past information, run short of space for stored document entries, or repeatedly move a recurrent state while generating an answer. Those pressures call for different changes.
The studies suggest a practical first question: which operation is costly in the intended workload? One changes attention retrieval, one chooses what a compressed cache retains, and one writes recurrent state less often.
Their reported outcomes belong to different tasks and measures, so each is useful within its own tested conditions.
What you will understand
Which operation does each method change: retrieving past tokens, retaining document entries, or writing recurrent state?
How do sampling for attention and rehearsal for cache selection make different choices about past information?
Which workload details and reported limits matter before applying a result to a serving system?
Three costs in one serving system
Imagine an assistant answering a question about a long document. To generate a response, its attention mechanism may need information from many earlier tokens.
An attention system can spend work searching among those tokens even when it reads only a selection. The vector search study asks how to choose that selection so the resulting answer remains useful.
Its resource target is the work of retrieving earlier information for the current question, not the amount of document information stored beforehand.
The same document may have been prepared before anyone knew the question. A system that stores many such documents can keep only a fraction of each document’s attention cache, the stored entries used when later questions refer back to it.
Now the decision comes earlier: which entries survive compression? The Attic study examines that choice when one compressed document must support later, unknown questions.
Spending less space on retained entries is a different aim from reducing search work for a question already in hand.
A third pressure appears as an answer is generated token by token. In models with a recurrent state, each new token normally triggers a read, an update, and a write of that state.
Moving it repeatedly can be expensive at serving batch sizes. The ReplaySSM study asks whether those writes can happen less often.
Together, the cases give a concrete decision to make before calling inference more efficient: is the immediate cost retrieval work, stored cache capacity, or repeated state writes?
Where the lesson goes next
Which past information to read and keep
When to write a new state
Match the evidence to the workload