A rule for splitting images can preserve object-finding recall
In plain words
Vision-language models often miss small symbols when asked to find them across a large drawing. This paper develops a theory and a rule for splitting such images into overlapping views.
The rule keeps each view detailed enough to show the targets and overlaps neighboring views so targets are covered.
In tests on construction drawings and floor plans, the rule did not significantly lower recall in any of eight model-and-data comparisons. Recall means the fraction of labeled targets the model found. The result applies to those tested comparisons, not every model or image.
The research
When a model misses small objects in a large drawing, dividing the image into views may help, but the views could also lose detections. The paper asks what conditions make splitting safe for recall, and how to choose views that cover the image without reducing the detail each target receives.
The authors derive when overlapping views can preserve expected recall, then give a rule that chooses view detail and overlap to meet those conditions.
The authors ask when splitting a large image into views can preserve a model’s ability to find its targets. Their theory says this depends on views keeping at least the whole-image detail per target, covering the targets, and on more detail and less content not hurting detection on average.
They compared their rule with whole-image calls on construction drawings and floor plans. Across eight model-and-data comparisons, the rule never significantly lowered recall and raised it significantly in four.
Since recall is the fraction of annotated targets found, these results support the rule under the tested conditions.
The recall guarantee depends on average monotonicity assumptions that may fail for some models and image distributions. The authors found high-resolution exceptions for tested OpenAI models, so their guarantees cover those models only on default paths; the theory does not guarantee F1.
Analysis
Deep
- Authors
- Junzhe Shi, Yuan Gan, Shida Jiang
- License
- arXiv non-exclusive