A checklist makes multi-reference image errors easier to locate
LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis · arXiv:2611.06856
In plain words
A picture made from many references can look convincing but still use the wrong detail, connect the wrong people or objects, or include something the instructions forbid. LongBanana is a test set that turns those requirements into separate visual checks, so errors can be located.
Its evaluator compares each check with the generated picture and relevant references. On the reviewed examples, it matched human labels exactly more often than a judge that made one call per image, though the test covers only 90 cases.
The research
A convincing image can still fail the instructions in ways a single overall impression may miss.
Separating content, relationships, and scene-wide requirements makes those failures easier to identify, and the audited comparison suggests that routing evidence to individual checks can bring automatic judgments closer to reviewed human labels on these cases.
LongBanana turns multi-reference image requirements into graph-derived visual checks, while LongBananaEval judges each check against its relevant references and records supporting evidence.
The researchers compared LongBananaEval with a single-call visual-language-model judge against reviewed human labels for 46,700 checklist decisions from 900 generated images. LongBananaEval matched the human label exactly on 66.81% of items, versus 50.99% for the single-call judge.
The two judges used the same backbone and item criteria, so the comparison tests how the checks are organized and evidence is routed. A separate double-label audit found 89.4% exact agreement between annotators on 7,005 judgments, supporting the use of reviewed labels as a reference for this pool.
The findings come from LongBanana’s 90 image-synthesis cases and its audited outputs.
The benchmark does not cover video, 3D scene composition, or unrestricted multimodal planning; creating its dense cases also requires costly expert annotation, and its evaluator depends on multimodal models and verification tools.
What the evidence shows
- A polished image can still fail
LongBanana is a benchmark for testing whether an image-generation system can combine many references into one image while meeting the task’s requirements. It treats correctness as more than visual polish: a system must choose relevant evidence, attach attributes and relationships to the right entities, avoid distractors, and satisfy scene-wide rules. Separate checklist items make these obligations inspectable.
- Requirements become separate checks
The benchmark represents each task with an entity–relation graph and derives visual checks from its entities, relationships, and scene-wide constraints. Experts design scenarios, an LLM structures them, and human curators review the resulting instructions, references, and checks. This makes the intended requirement and the evidence needed to judge it explicit.
- The evaluator agrees more often
Because both judges used the same backbone and item criteria, the higher exact-match rate suggests that organizing judgments around checklist items and routing relevant evidence improved agreement with reviewed human labels in this audited pool.
The separate annotator check indicates that those labels had measurable, though imperfect, consistency.
- An adaptive workflow scores higher
This comparison suggests that the adaptive workflow achieved a higher checklist average than an optimized fixed schedule on these cases, even though the fixed schedule received the construction graph and the agentic workflow did not.
Because their access to task structure differed, the result describes these tested systems and setup rather than isolating the workflow alone.
- The stages do not help every task
A five-configuration comparison held the OpenClaw harness and image generator fixed. The combined workflow led on its difficulty-weighted overall score and on the share of tasks reaching 70 points, but the minimal harness scored higher on temporal and state reasoning.
The stages therefore help in combination on the aggregate measures without improving every task category.
Deep
- Original paper authors
- Haoxiang Cao, Yuxuan Zhang, Penghui Du, Bo Li, Jiajiong Cao
- Submitted to arXiv
- License
- arXiv non-exclusive