Math overlap rules can remove many distinct problems

When math training data are screened against test questions, similar problems may share a template without sharing an answer or being copies.

It remains unclear whether broad matching rules catch test exposure at the cost of removing legitimate practice problems, and how much that choice costs in training data.

The paper compares eight existing removal rules and finds that case-by-case adjudication better balances reliability and coverage on held-out candidate pairs.

To remove test exposure from math training data, how can curators avoid deleting distinct problems that merely share a familiar setup?

This paper compares eight existing removal rules against independently reviewed training–test pairs, with a separate item review to check whether flagged items were actually safe to remove.

No tested similarity cutoff met the preregistered reliability target on calibration pairs. On held-out pairs, an existing adjudicated rule scored highly on both measures: resolved precision is the share of flagged pairs confirmed as matches, excluding uncertain pairs, while candidate-frame recall is the share of confirmed matches it caught among pairs surfaced by the pipeline.

These results support the authors’ view that reviewing individual matches can balance the measures better than a single similarity cutoff in this pipeline.

These findings apply to the reviewed candidate pairs from the audited English-language math corpora and specific test snapshots. The small item sample does not establish population-wide false-removal rates, and the study did not measure whole-corpus recall or downstream training effects.