A reusable cache keeps tabular predictions accurate after compression

When a tabular model answers many queries from the same labeled examples, what can be compressed without losing too much predictive accuracy? A fixed smaller example set is reusable but may omit useful rows, while choosing examples separately for each query limits reuse and raises serving cost.

The paper asks whether compressing the model’s stored attention information can keep one memory useful across queries.

QCOC compiles a fixed example set’s attention cache into reusable memory, then fits its values to preserve the original cache’s responses.

The researchers compare QCOC with direct weighted cache compression at the same memory sizes, using TabICL v2 to predict held-out rows on 64 classification datasets. QCOC’s mean accuracy was higher at both sizes, while full context remained higher on average.

Across a separate grid of longer contexts, QCOC led among compressed methods in most settings and stayed close to full context on average.

Together, the comparisons support compressing one reusable memory while retaining useful predictive information, with some settings favoring query-specific retrieval.

The fitting queries come from the in-context examples, and the theoretical error bound depends on stated assumptions; neither guarantees accuracy on new queries.

Compilation also has a substantial one-time cost, and one shared memory can miss query-specific structure, as the Covertype results illustrate.