VisSearch-Bench tests how agents connect images and searches
VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents · arXiv:2611.04894
In plain words
VisSearch-Bench is a test of whether software that can inspect images and search the web can connect visual clues from one step to the next, then use them in an answer. Its tasks include checking sets of objects, following clues, and telling similar-looking candidates apart.
The test scores fully correct answers separately from required findings recovered along the way. For one tested model, a search agent scored higher on both than answering from the question and images alone. The results describe this test's tasks and setup.
The research
A polished final answer can conceal a broken chain of evidence: a system may miss a visual clue or fail to carry it into a later search.
Separating complete answers from intermediate findings changes what a score can tell us about this specific ability, while keeping conclusions tied to the tasks and systems actually measured.
VisSearch-Bench is a test of whether multimodal agents can link visual evidence across open-web searches, separating full-task success from recovery of intermediate findings.
Annotators made 204 public-source tasks with documented evidence and checks against visual shortcuts.
A task-specific evaluator scored each system's first complete answer for full correctness and for recovery of required intermediate findings, averaging the fraction recovered within each task across tasks.
For one Qwen model, ReAct scored higher on both measures than direct answering. Across systems, the top complete-answer score and top intermediate-recovery score came from different systems, showing the measures capture distinct outcomes.
Each system gave one first complete response per task from a fresh context, under fixed inference budgets. The findings describe those systems on these 204 tasks; repeated-run variation was not reported.
What the evidence shows
- Two kinds of search success
VisSearch-Bench is a benchmark for testing whether multimodal agents can connect visual evidence across images and open-web searches. It scores complete-task Accuracy, the share of tasks answered fully and correctly, and Process Score, the average across tasks of the fraction of required intermediate findings recovered within each task. Those measures separate finishing a task from recovering useful parts of its answer.
- Tasks require linked clues
The tasks cover different ways visual evidence can guide a search: checking a set of objects, following clues from one step into another, combining broad and step-by-step searches, distinguishing similar-looking candidates, and using image structure such as position or order.
Their answers are grounded in public sources, with documented evidence and checks for shortcuts around visual requirements. This makes the test about connecting visual and external information, not simply using an image somewhere in a task.
- Search improved one model's scores
For Qwen3.5-397B-A17B, ReAct scored higher than answering without tools on both measures. This paired comparison shows that search coincided with more complete answers and recovery of required intermediate findings for this model on these tasks.
- The top score depends on measure
The system with the highest score depends on what is measured: ReAct with Kimi-K3 led in fully correct tasks, while ReAct with GPT-5.5 led in recovered findings. This difference shows why partial recovery and complete-task success describe distinct outcomes across the tested systems.
Deep
- Original paper authors
- ZhengXian Wu, Qianhao Nie, Zhuohong Chen, Junjie Gao, Yuyao Zhang
- Submitted to arXiv
- License
- arXiv non-exclusive