Why a shared prompt cache raised tail latency

A shared prompt cache is useful only if retrieving saved prompt data costs less time than recomputing it.

In this production workload, enabling the remote tier left median time to first token unchanged but sharply raised the p99 for 16k-token requests, leaving the practical question of why a cache hit could slow serving.

The paper models when remote cache hits beat recomputation, then shards cached data across GPUs to reduce stalls while preserving high cache use.

The production result raises a concrete question: why did a shared cache intended to avoid repeating prompt work make the slowest 16k-token requests slower? The author models cache-hit time as transfer plus fixed overhead, then compares the required bandwidth with what one remote store delivered.

The model’s break-even requirement exceeded the measured bandwidth, supporting the explanation that hits could lose to recomputation on this setup. A store split across ranks then spreads cache work.

In one hit-heavy replay, the full design had much lower p99 prefill latency than synchronous loading through one instance, while both served 98.5% of prompt tokens from cache.

The latency replay was run once on an isolated, prefill-only engine, using one prompt-length bucket and one engine as the cache client. The separate end-to-end deployment comparison changed other deployment conditions too, so it does not isolate the cache design’s effect.