Short bursts can leave agent-serving caches degraded
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving · arXiv:2611.13412
In plain words
A language-model server can reuse parts of an agent’s earlier prompts to answer later turns faster. The paper studies what happens when many sessions briefly arrive at once, and introduces a gateway controller that can make active sessions wait when their estimated memory demand is high.
In paired vLLM runs with the same base arrivals, 9 of 12 burst runs stayed degraded after the burst, while none of their no-burst twins alone did. The tests suggest that controlling sessions already in progress can help restore or preserve cache use in the tested settings.
The research
A server that stays slow after a rush has passed can keep accumulating unfinished sessions even though its arrival rate is back to normal.
These results change the recovery question: limiting new sessions alone may not be enough; managing sessions already in progress can matter when cache demand sustains the slowdown.
The study traces how a brief burst can leave agent-serving caches degraded, then tests gateway controls that park active sessions to recover or prevent that state.
In paired vLLM runs, each burst arm and its no-burst twin received the same base sessions; only one received five minutes of extra arrivals. Persistent degradation appeared only in the burst arm in 9 of 12 pairs, while no pair degraded only in the twin.
A cache-disabled control’s backlog shrank after its burst, pointing to a cache-related regime rather than unfinished burst work alone.
The authors link low cache use to a feedback loop: slower turns leave more sessions active, their combined memory demand strains the pool, and uncached prompts take longer to recompute.
After collapse, a gateway budgeter that parks sessions by estimated demand restored cache use in all seven matched runs, supporting active-session control as a recovery approach in this setup.
The experiments used one GPU type and one replica, with finite observation windows and no production traffic. The authors infer that prefixes are evicted between turns because the engines do not report individual eviction events.
What the evidence shows
- A burst can outlast itself
The study examines a persistent failure in language-model agent serving: after a short arrival burst, cache hit ratio and throughput can remain low even when the server returns to its prior base arrivals.
In paired vLLM runs, the no-burst twin shared those base arrivals with the burst run, so the comparison tests whether the temporary extra arrivals changed the later outcome.
- The paired runs separate histories
In 9 of 12 pairs, only the burst run persistently degraded; no pair had degradation only in its no-burst twin. Two pairs degraded in both arms and one in neither.
The burst therefore increased the likelihood of entering the degraded regime, but spontaneous degradation in some twins means it was not required.
- The cache changes the aftermath
A no-cache control helps distinguish the persistent cache-enabled regime from a burst backlog that simply takes time to clear. With caching disabled, all four absolute-burst runs had a shrinking or already-restored backlog in the final 30 minutes.
In a smaller rate-proportional burst test, all four pairs ended with only 0–5 unfinished sessions per arm.
- Low cache use sustains the loop
In degraded runs, nearly all completed turns had less than half their prompt cached, and the token-weighted hit ratio was 0.10–0.12. The authors interpret this as evidence for a cycle in which slow turns keep more sessions active, their memory demand strains the shared pool, and missing prefixes make turns slower.
The engine does not expose individual eviction events, so the specific between-turn eviction step is inferred.
- Parking helps recovery
Among seven matched runs already collapsed in every recovery arm, a budgeter that parks sessions by estimated memory demand recovered all seven in under five minutes. An admission-only cap, which limits new sessions but never parks admitted ones, recovered one of seven.
This contrast supports controlling active sessions for recovery in the tested setup; separate prevention runs also found no persistent degradation under their listed triggers.
Deep
- Submitted to arXiv
- License
- CC BY 4.0