Where AI inference spends memory and time
Which bottleneck does a change actually reduce?
Keeping fewer tokens, reading less cache and generating several tokens at once change different parts of inference. Identify the measured resource before comparing a memory reduction with a speed improvement.
- Keep less cache
- Memory stored · eviction
- Read less cache
- Memory traffic · sparse attention
- Generate in blocks
- Sequential steps · drafting and verification
Three questions to take into your reading
- Does the result concern stored memory, memory traffic or computation?
- Is the timing for prefill, decode or the whole request?
- Which hardware, context length and task quality accompany the result?
Follow the evidence
Match cache changes to the cost your workload pays
Connect cache changes to the resource a workload pays for.
A smaller cache can save stored memory, while selective reads can reduce the information fetched for each new query. Removing visual entries during answer generation changes a third cost. A quality score is useful for choosing among these approaches only if its test exposes the phase being changed. The studies point to a practical first step: identify the constrained resource, then test the corresponding method on the intended workload. Their classification, language-modeling, and visual-answer results describe different outcomes, so they do not yield a shared ranking.
DropKV leads tested eviction methods with a small token cache
Inspect a method that removes cached tokens under a limited cache budget.
DropKV improves average long-context task scores over tested eviction methods on three models at a 5% cache budget.
Score-guided sampling improves sparse attention
Compare selective cache reading with permanently removing stored tokens.
Stratified Sparse Attention uses score-guided sampling to improve sparse attention results in selected long-context language-model tests.
A coordinated system makes low-bit language models generate in blocks
See how generation steps, low-bit execution and runtime are evaluated together.
BitMTP coordinates low-bit block drafting and runtime execution, raising measured generation speed while keeping tested mean scores close to baselines.
Where the conclusion stops
Speedups from different workloads cannot be added or ranked as if they came from one experiment.