Disaster damage scores drop when a whole event is held out
In parole semplici
A model can seem good at spotting disaster damage if it is tested on tiles from disasters it has already seen. This paper audits image features from four models to see whether they predict damage on a disaster left out of training.
The researchers found that average scores fell when they tested on a whole unseen disaster, across two image collections. That means familiar-disaster scores did not show how well the models transferred to new events in these tests.
La ricerca
When a model predicts damage on tiles from disasters it has already seen, is it recognizing evidence that will transfer to a new event?
Tiles in one disaster can share geography, sensors, building density, and damage prevalence, so a random split may let familiar-event clues make the test look easier.
The paper audits fixed geospatial image embeddings across disaster events, then tests source-only feature changes for predicting damage in an unseen event.
The question is whether strong scores on familiar disasters indicate damage evidence that transfers to a new event. Tiles from one disaster can share scene and event clues, so a random tile split may not answer that question.
The authors compare random splits with tests that hold out a whole event. For full-pair features, mean balanced accuracy, averaging recall for damaged and undamaged tiles, falls from 0.807 to 0.638 on xView2 and from 0.818 to 0.568 on BRIGHT.
This supports their conclusion that familiar-event scores overstate new-event performance in these tests.
The results concern tile-level damage labels and the tested image pipelines. The authors leave building-level severity and native sensor inputs for future evaluation; the findings do not establish reliability on other data or tasks.
Analisi
Dettagli
- Autori
- Ali Shibli, Andrea Nascetti, Yifang Ban
- Licenza
- arXiv non-exclusive