A code checker finds silent prompt-caching defects
Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents · arXiv:2610.34817
In plain words
Some applications use prompt caching to reuse work when the beginning of a prompt stays exactly the same. Small integration mistakes can stop that reuse or make its usage count wrong, even while the model still gives a correct answer.
The paper introduces CacheLint, a code checker that looks for missing safeguards around caching calls. In 571 repositories beyond the projects used to build its rules, 32 of its 37 warnings were confirmed as real defects; that result measures precision, not how many total defects it found.
The research
When an application integrates prompt caching incorrectly, the model may still answer normally even as caching stops working or usage counts become wrong. How can developers find such defects when ordinary errors provide little warning?
CacheLint checks caching call sites against provider support and code safeguards, flagging application defects that can go unnoticed while answers remain correct.
The authors built CacheLint, a static code checker that looks for missing conditions around prompt-caching calls. It uses CodeQL to locate sites, a dated provider-capability lookup to check provider requirements, and an LLM to inspect nearby code for safeguards.
On 571 repositories outside rule development, researchers confirmed 32 of 37 warnings as defects, for 0.86 precision. This shows CacheLint can identify real defects in a broader repository set, but without complete defect labels the study cannot say how many it missed.
The 0.78 recall comes from the same 120 fixes used to derive CacheLint’s rules, so it is an in-sample result. The broader repository set had no complete defect labels, and CacheLint cannot check defects that require comparing prompts across calls or turns.
What the evidence shows
- Where caching breaks
The paper studies application-side defects in prompt caching, which reuses previously computed work when a request begins with an identical prompt prefix. Provider requirements differ and can change, while applications may also alter the prefix or misread returned usage counts.
The authors examined merged fixes from ten selected open-source agent applications to learn what goes wrong and build a detector.
- Why errors miss defects
The study found 97 of 120 defects corrupted outgoing requests, while 23 misread cache-usage counts without changing the request. In 108 cases, the defect failed silently: answers stayed correct although cache reuse or its reporting was wrong.
Since ordinary runtime errors often gave no signal, the authors developed a code checker that looks for missing conditions at caching call sites.
- A checker for call sites
CacheLint locates caching call sites and checks whether conditions for correct use are met. CodeQL finds the sites; a dated provider-capability lookup checks provider support, and an LLM examines surrounding code for safeguards such as keeping a prefix stable.
In the ten applications used to derive the rules, 94 of 132 warnings matched labeled defects, while 26 of 120 defects were missed.
- A broader repository check
The authors also applied CacheLint to 571 repositories excluded from rule derivation. They manually confirmed 32 of 37 warnings, a precision of 0.86, but could not measure recall because the repositories had no complete defect labels.
When either specialized check was removed, precision fell on this same set, supporting the role of both checks in filtering benign sites.
Deep
- Original paper authors
- Ying Zhang, Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li
- Submitted to arXiv
- License
- CC BY 4.0