In plain words
A model can use new examples in its temporary context, or training can write information from those examples into its weights. This paper introduces a way to compare those choices by checking how much each helps the model describe the answers to new questions.
In one trained-transformer evaluation, the method issued a choice in some comparisons, and every issued certificate matched the measured winner. When the examples fit the tested context window, the cache did better than the selected full-training recipes.
The research
When a model receives new examples, should it use them in temporary context or should training write them into its weights? Context is rebuilt for prediction, while training changes the model’s stored state. The paper asks how to compare these routes using the predictions they make on new queries.
The method compares how many bits held-out predictions save from context and trained weights, then uses capacity limits or a held-out test to choose between them.
To compare context with weights trained on the same examples, the method measures how much each makes probability-based descriptions of new queries shorter, then checks a channel’s gain against the other’s capacity limit.
In a trained three-layer transformer, certificates were issued in 25 of 64 comparisons, and every call matched measured winners. When examples fit the tested cache window, selected full fine-tuning recipes left the weights unchanged and the cache saved 2.7 to 7.0 more bits per query block.
The certificates require exact information terms, which the paper has for designed distributions; when those terms are unknown, it uses a held-out-query test instead. Agreement in the reported comparisons does not establish that certificates always decide correctly in other settings.
Analysis
Deep
- Original paper authors
- Leon Chlon, Julie Huang
- Submitted to arXiv
- License
- CC BY 4.0