A lazy-pulling cache keeps full disks from breaking model reads
Why Kubernetes Cannot Contain a Full Lazy-Pulling Cache: Two Impossibility Results and a Bounded, Honest Design for Model Serving · arXiv:2611.11739
In plain words
A lazy-pulling cache downloads model-file pieces only when a program needs them, then saves those pieces on the machine. But if its disk fills, a cache write can also interrupt the program’s read.
The paper changes the cache so it can skip saving a piece when storage fails while still delivering the fetched data. On one tested model image, the updated system read all 280 files after the cache was 90% full; the older version failed on 273 files.
The research
A green pod status can hide failures in the model files that pod is serving, while disk eviction may target other workloads instead of the cache. This result changes what a cache design must demonstrate: it needs to bound the storage it owns and keep a failed cache write from interrupting a read.
A bounded lazy-pulling cache keeps cache-write failures from interrupting model reads, and its full-read test stayed error-free under heavy cache pre-fill.
On a two-host rig, the researchers read a 280-file model image through upstream stargz-snapshotter and the proposed cache after filling the cache to 90%.
Upstream failed on 273 files after 4.1 GB; the proposed system read 150.3 GB with no errors in both replicates, while skipping thousands of cache writes.
That result shows the cache-write failure no longer vetoed delivery on this workload. A separate comparison found the full read took a median 18.5% longer at 90% than at 0% pre-fill across five paired runs.
The full-read comparison used one 280-file workload on a two-host rig with a same-availability-zone registry. The design is opt-in and off by default, and the evaluation does not establish the same results for other workload shapes or registry locations.
What the evidence shows
- A hidden read failure
The paper introduces a design and implementation for lazy-pulling caches in Kubernetes model-serving nodes. The central problem is that a full cache can make model-file reads fail even while a pod’s readiness, health and prediction signals remain green.
The question is whether monitoring or platform eviction can contain that failure, or whether the cache itself must manage its disk use and read path.
- A green probe missed damage
The sentinel readiness probe reported green while 272 of 280 model files were unreadable. That test shows why a probe that samples selected files cannot guarantee that the rest of the model is readable.
The probe result makes monitoring an insufficient answer to the tested failure, even when it reads through the model mount.
- Eviction targeted the wrong bytes
Kubelet raised a disk-pressure signal when the cache partition was full, yet reclaimed zero cache bytes and evicted 26 pods that had never written there. The contrast is between detecting pressure and having an accounting path that lets the platform reclaim the affected bytes.
Here, the cache sat outside containerd’s image accounting, so platform eviction did not contain the cache failure.
- The cache owns its limit
The paired failure test shows what the design changes: a cache-write failure no longer prevents already-fetched bytes from reaching the reader. On the tested workload, the proposed system completed the full read without errors, supporting this design under that cache-pressure condition.
- The answer has a cost and scope
The full-read test shows the design traded extra completion time for continued file access under cache pressure. At 90% pre-fill, the proposed system’s median completion time was 18.5% longer than at 0% across five paired runs. This cost is bounded to the tested rig, which used a registry in the same availability zone.
Deep
- Original paper authors
- Georgii Kliukovkin
- Submitted to arXiv
- License
- arXiv non-exclusive