A method separates data selection from support shifts
In plain words
Researchers want to know whether a training-data filter helps because it picks better examples from each kind of data, or because it changes how much of each kind is used. Comparing the filter with a random sample mixes those two effects.
The paper introduces a method that builds matched training sets to separate them. In one Korean, English retrieval test, selection within data groups stood out against a random-mask reference; the test did not settle whether changing the group mix also mattered.
The research
When a training-data filter is compared with a random sample, both the examples selected within data regions and the regions’ shares change. Which change accounts for a downstream difference? Separating them matters because selection and the data mix are distinct parts of what the filter does.
CGSD compares gate-eligible and unrestricted examples at matched support shares and token budgets, separating within-region selection from shifts in the data mix.
A filter-versus-random comparison changes both which examples are selected within data regions and how much each region contributes. Can the downstream difference be separated into those effects?
CGSD compares four training sets at a matched token budget, crossing eligibility with support shares. In Korean–English retrieval, within-support selection averaged +29.7 holdout items, above a +14.0-item reference calibrated with same-rate random masks. Redistribution and interaction remained unresolved.
These retrieval results cover one backbone, endpoint and data family. Redistribution and interaction remain unresolved, and the mathematics findings concern whether the design can be built, not whether the gate changes performance.
Analysis
Deep
- Original paper authors
- Hyojung Han, Jongmin Kim, Seung-Hun Jeon
- Submitted to arXiv
- License
- CC BY 4.0