Synthetic night images change test-set model rankings
En palabras sencillas
A test set can give an incomplete picture if it has too few images from a condition, such as night, that changes how a model performs.
This paper introduces a pipeline that finds important missing conditions, changes existing images to represent them, and checks the new images before adding them to the test set.
In two day/night tests, adding generated night images changed which of two models the test set favored. The result matched tests that included real night images, but it applies to the tested classification and detection setups.
La investigación
If a test set lacks a condition that affects model performance, could adding images of that condition change which model evaluators judge to be better? The challenge is that evaluators may not know which conditions are missing or which ones matter when they cannot inspect the training data.
The authors build a three-stage image pipeline that finds important missing conditions, generates matching images, and checks them before adding them to test sets.
The pipeline labels day and night by matching images with text descriptions, then checks whether those labels cluster in a task model’s image representations. It transforms covered images into night images and checks their similarity and dataset coverage before inclusion. Evaluators compared two task models on original, synthetic-augmented, and real-night test sets.
On both datasets, original tests ranked the day-only model higher; augmented and real-night tests ranked the day-and-night model higher.
Because the augmented sets agreed with the real-night reference, they changed model selection in the same direction in these tested classification and detection tasks.
The evidence comes from two deliberately imbalanced day/night test setups, and the authors’ additional BDD100k weather tests found less accurate importance filtering for subtler conditions.
Global embedding measures can also miss fine spatial details needed for object detection, so these results do not guarantee that synthetic images will reproduce real-image performance in other tasks.