Match cache changes to the cost your workload pays
A smaller cache can save stored memory, while selective reads can reduce the information fetched for each new query. Removing visual entries during answer generation changes a third cost. A quality score is useful for choosing among these approaches only if its test exposes the phase being changed.
The studies point to a practical first step: identify the constrained resource, then test the corresponding method on the intended workload. Their classification, language-modeling, and visual-answer results describe different outcomes, so they do not yield a shared ranking.
What you will understand
How can you tell whether a cache method changes stored size, information read per query, or entries retained during decoding?
What does each study's reported quality measure, and what is its comparison?
Why must a test score later generated tokens to assess decode-time visual cache eviction?
Locate the cache cost
Imagine serving many predictions from one labeled table. Keeping all its examples available can make repeated predictions costly.
In a language model, earlier words occupy a cache that each new word may read. In an image question-answering model, visual entries can remain in that cache while the answer is generated.
Each situation invites compression, but the costs arise at different points. The useful question is whether stored size, repeated reading, or retained visual information is the constraint.
For the table, the labeled examples form a context reused across queries. Selecting a small fixed set of rows makes that context reusable, but may leave out rows useful to a particular query.
Selecting rows anew for every query can improve local relevance while limiting reuse. The table study takes a third route: it compresses the internal cache built from the examples into a smaller representation that can still give different responses to different queries.
The other two settings make the choice less like selecting rows. A language-model query may need detail from different parts of a previously stored key–value cache, the model's record of earlier inputs.
An image question-answering model first processes its prompt, called prefill, then generates later answer tokens in decoding. It can reduce work during prefill or remove visual cache entries before decoding. Which resource should a practitioner target, and what test could reveal the resulting quality cost?
Where the lesson goes next
Change the representation, the read, or the retained entries
Read each quality result in its own setting
Test the phase and outcome that matter