Chiosa
Systems & Infrastructure

Reusable AI session context can strain GPU memory

2 min
Published on Chiosa Updated
#infrastructure#inference-memory#idle-eviction#real-world-deployment

In plain words

AI sessions often make several model calls while building up the same conversation context. The saved calculations for that context can speed later calls, but storing them uses GPU memory even while a session is idle.

The authors studied four Microsoft M365 workloads and replayed their call records. In one Research replay, removing saved context after an hour greatly reduced estimated memory use, while only slightly increasing the share of calls that had to rebuild context. The result depends on the study’s memory estimates and replay rules.

The research

When a model session makes repeated calls, saved context can avoid recalculating information, but keeping it in GPU memory uses capacity even while the session is idle. How much of that memory can be reclaimed before many returning calls must rebuild their context?

The authors replay production call traces to test how idle eviction, recoverable cache copies and deployment-local state affect context rebuilding.

The authors replayed production call traces to test whether saved session context can be reclaimed without making many calls rebuild it. Because production model layouts are confidential, they estimated KV memory from a public reference layout and compared residency rules on Research traces.

In the complete eligible Research trace, a one-hour idle cap reduced estimated peak fleet residency from 501.8 TB to 140.8 TB, while the share of calls requiring reconstruction rose from 4.07% to 4.53%.

This supports the view that retaining peak state through long idle gaps inflates the estimate, while the tested cap adds relatively few reconstruction calls.

These are estimates and counterfactual replays, not measurements of a deployed residency policy. The traces cover four Microsoft M365 workloads, and the replay does not measure how lower-precision cache copies affect responses, latency or transfer costs.

Analysis

Deep

Original paperUnderstanding KV Residency in Production-Scale Agentic LLM WorkloadsOpen original
Submitted to arXiv
Related paperSystems & InfrastructureWhy a shared prompt cache raised tail latency#infrastructure#inference-memory#gpu-streamingRelated paperSystems & InfrastructureDifferent retry limits help protect important service requests#infrastructure#real-world-deployment#differential-retriesRelated paperSystems & InfrastructureA GPU tokenizer preserves IDs while speeding cached requests#infrastructure#gpu-streaming#merge-latencyRelated paperSystems & InfrastructureEmber keeps oversized GPU workloads moving with adaptive placement#infrastructure#inference-memory#gpu-streaming
Next paperWhy a shared prompt cache raised tail latency