In plain words
Language-model agents can build long histories as they work. Shortening that history by editing its text can mean rebuilding cached states, while summaries can lose details.
This paper introduces cache engineering: an agent can drop selected cached states while keeping its written history and reusing the rest.
In one test of GPT-OSS-120B on 100 browsing tasks, direct cache dropping was reported to lower cost and improve accuracy; that result applies to this tested setup.
The research
When an agent works through a long task, shortening its written history can force the model to rebuild cached information, while summarizing may remove details needed later. Can an agent reduce what the model attends to without rewriting its history or giving up useful information?
The authors introduce cache engineering and AKV, letting agents drop selected cached states while preserving their text history and reusing retained states.
The authors introduce cache engineering, a way for an agent to drop selected cached states from future attention while keeping its text history intact. AKV is the serving system that tracks these operations so retained states can be reused correctly.
In a test of GPT-OSS-120B on 100 browsing tasks, text trimming cost more than full-context inference, while threshold-triggered summaries reduced cost but lowered accuracy.
Direct cache dropping was reported to improve accuracy and reduce cost, supporting the idea that cache control can ease this trade-off in the tested setting.
These results cover the tested models, tasks and serving workload. The serving comparison used different concurrency levels and replayed trajectories collected under each method, so it does not isolate cache management from every workload difference.
Analysis
Deep
- Original paper authors
- Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li
- Submitted to arXiv
- License
- arXiv non-exclusive