Match overlap evidence to the decision it will change

A repair model may have seen the patch it is asked to produce. A pooled brain-scan repository may give one person two identifiers.

A math training problem may resemble a benchmark question while posing a different task. Each case creates pressure to act on a possible overlap, and each makes a mistaken decision costly.

The evidence needed for that decision depends on what is being compared. The studies use code history, reviewed images, and judgments about problem content to move from possible matches toward conclusions within their tested settings.

Imagine scoring a model on a collection of past software bugs. Each task gives the model a defective codebase and asks it to produce a repair; the collection also contains the developer’s patch for checking the task.

A successful repair is harder to interpret if that patch appeared in material used to train the model. The score could then reflect some familiarity with the answer as well as an ability to solve the bug.

This concern is concrete for benchmarks drawn from public repositories. Code training collections can contain repository snapshots taken after a patch was merged, so a benchmark’s answer may remain in a later copy of its source project.

Yet finding the repository in a public collection does not reveal what an undisclosed model actually saw. An auditor needs evidence about both the project and the particular patch before judging the benchmark result.

The shared question is how much evidence should move an audit from a possible match to a decision about data. For a repair benchmark, comparing the model’s final patch with the developer’s patch is an imperfect clue: a model might work out a similar fix, or produce a different fix after learning the original idea.

The code study therefore investigates exposure without relying on the final repair as a direct test of memory.