A trial-aware image test controls false matches
Identities, Not Pairs: Distribution-Free False-Accept Control for Embedding-Based Verification · arXiv:2610.35277
In plain words
The paper introduces COVE, a method for deciding whether two images show the same identity. A common approach sets a threshold using scores between calibration images, but those pairs share identities, so one unusually similar identity can affect many scores.
COVE ranks the trial pair alongside the calibration pairs and pairs connecting the trial images to calibration images. When identities are sampled comparably and the same symmetric scoring rule is used, the method’s false-match rate stayed close to its mathematical bound in the paper’s random-split image evaluations. The guarantee averages over calibration samples and trials.
The research
When image matching must limit false matches, a standard threshold recipe counts scores from many calibration pairs. Because identities recur across pairs, those scores are dependent.
It remains unclear whether a method can control false accepts without assuming those pairs act like independent evidence.
COVE ranks each trial alongside calibration and trial-linked pairs, giving a finite-sample marginal false-accept bound under identity exchangeability.
COVE ranks a trial pair among the calibration pairs and the pairs connecting its images to calibration images. Under identity exchangeability and symmetric scoring, its rank gives a finite-sample upper bound on the marginal false-accept rate, averaged across calibration samples and impostor trials.
In random identity-split evaluations, the realized rate across 903 cell-and-target comparisons had a median of 1.001 times the exact bound, with no statistically significant deviations under the stated test.
This supports the theorem in those evaluated settings, while leaving fixed-sample conditional control unresolved.
The guarantee is an average over calibration draws and impostor trials, and requires calibration and test identities to be sampled comparably with the same symmetric scoring rule. In tested Stanford Online Products splits that broke this condition, COVE’s realized false-accept rate ranged from 0.50 to 2.48 times its target.
The result does not cover a fixed calibration draw, targeted look-alikes, or face and person verification.
What the evidence shows
- The shared-identity problem
The paper introduces COVE, a method for deciding whether two images show the same identity while controlling false matches. A usual calibration recipe chooses a threshold from scores between different identities.
But those pairs are dependent because each identity appears in many pairs, allowing one unusually similar identity to influence many scores together.
- A limit on calibration alone
The paper first establishes a limit for thresholds-based only on calibration images. Across exchangeable impostor populations with a hub identity, any such rule that controls false accepts across those populations has limited probability of reaching the hub score level on a hub-free calibration sample. This is a worst-case theorem, not a measurement about a particular encoder.
- Rank the trial with its context
COVE includes the probe and reference in the set of units being ranked. It counts calibration pairs and pairs linking either trial image to calibration images that score at least as high as the trial pair, includes the trial itself, and compares the resulting rank fraction with the target.
Under identity exchangeability and symmetric scoring, the theorem bounds the marginal false-accept rate by the target.
- The bound matched evaluations
In random identity-split evaluations, researchers compared the realized false-accept rate with COVE’s exact theoretical bound. Across 903 cell-and-target comparisons, the median rate was 1.001 times the bound, with no statistically significant deviations under the paper’s multiple-comparison test.
This evidence checks the marginal guarantee in the tested settings; it does not establish control for one fixed calibration sample.
- Pair independence misled controls
Pair-based controls showed why dependence matters in practice. The empirical quantile rule exceeded 1.1 times its target in 36% of eligible cases.
For a different rule that treated pairs as independent and claimed conditional control, the median share of calibration draws exceeding its target was 42% at calibration sizes of at least 3,000, compared with a nominal 5%.
Deep
- Original paper authors
- Salim Khazem
- Submitted to arXiv
- License
- CC BY 4.0