A model’s cue sensitivity can change which model is best
Cette page n’a pas encore de traduction vérifiée. La version anglaise est affichée. Lire la version anglaise
En termes simples
The paper introduces Shift, Reliance Alignment (SRA), a method for choosing a vision, language model when image questions change between a labeled source set and a target domain.
It checks each candidate on labeled images, measures whether misleading scene clues or text increase its errors, and estimates what proportion of target questions each cue misleads on.
On new misleading questions from OpenImages and Objects365, SRA selected a model with lower pooled error than choosing by source error alone. A new COCO test did not establish a reduction, so the result depends on the tested domain.
La recherche
A model chosen from one test set may not be the one with lowest error where misleading image cues are common. Can a selector predict that change without running every candidate on target images?
SRA uses labeled target calibration items to measure a cue detector’s error rates, but does not run the candidate models on target images.
SRA ranks image-question models by source error plus measured cue sensitivity weighted by estimated misleading-cue frequency in the target.
The authors measured each candidate’s error on labeled source images and its extra error after edits made scene context or text misleading, compared with matched control edits.
A separate cue detector estimated how often those cues misled in unlabeled target images; SRA combined those estimates with source error to choose a model.
On prospective tests with new misleading images from OpenImages and Objects365, the selected model’s pooled error was 18.8%, versus 30.4% when selection used source error alone. On new COCO images, errors were 6.4% and 6.6%, a statistically tied result.
This supports SRA for the tested shift in two corpora, while showing that its predicted advantage did not hold across all tested domains.
SRA assumes cue effects add together and that cues trigger the detector at the same rates in target and calibration samples. The study tests yes/no object presence, not attributes or counts; its new COCO test did not establish an error reduction.
Analyse
Détails
- Auteurs de l’article original
- Faisal Mohammad, Mohamed Younis
- Soumis à arXiv
- Licence
- arXiv non-exclusive