Small samples predict large image retrieval scores
In plain words
Image retrieval tests whether a system can put pictures of the same object or place near a query picture, ahead of unrelated images. Huge sets of unrelated images make the test more discriminating, but encoding them for every model takes substantial work.
The paper’s method samples some unrelated images, estimates how often they would outrank matches, and predicts the score for a much larger gallery.
On ILIAS, a 5% sample of 100 million distractors predicted the score for match rankings in the top 1,000 within 0.005 points on average across five tested backbones.
The research
When image retrieval tests use huge pools of irrelevant images, each model variant must encode the pool to measure how well it ranks relevant images. Could a small random sample support a credible prediction of the full-pool score while keeping the demanding gallery size?
The method estimates how often distractors outrank relevant images, then uses those chances to predict retrieval scores for a much larger gallery.
On ILIAS, a test of retrieving the same instances among distractors, the authors compare predicted mean average precision within the top 1,000 results per query with the score computed over all 100 million images.
From 5 million randomly sampled distractors, predictions differ by 0.005 mAP points on average across five backbones, with a maximum error of 0.008.
At the same sample cost, direct subset evaluation has 0.113 points mean absolute error. The tail-aware estimate better recovers the full-gallery ranking score by accounting for distractors that can outrank relevant images even when the sample does not show them.
The estimate assumes the sample is drawn uniformly at random from the distractor pool, and its tail extrapolation can be wrong if the fitted score distribution is unsuitable. The shard check can reveal differences within a sample, but does not prove that it represents the omitted images.
Analysis
Deep
- Original paper authors
- Boaz Meivar, Shai Avidan
- Submitted to arXiv
- License
- arXiv non-exclusive