A 3D test reveals models struggle with complete color counts
Where Do Vision Language Models Fail at 3D Reasoning? · arXiv:2610.39639
In plain words
BedBench is a test of whether vision-language models can reason about generated 3D shapes. It asks questions such as how many blocks of each color an object contains, and checks whether every part of an answer is right and related answers agree.
On the task where every color count had to be correct, the best model got the complete answer right on 7.2% of cases, compared with 93.3% for the human reference. Training one model further improved its score on that task, but did not settle whether it could use its counts consistently.
The research
Can a vision-language model answer separate questions about an object yet fail to give one complete answer, or make related answers agree?
BedBench tests this distinction using generated 3D shapes with known answers, so researchers can compare individual judgments, complete answers, and consistency on the same objects.
BedBench links exact-answer, consistency, and image-dependence checks on generated 3D objects to diagnose where vision-language model answers break down.
Researchers tested vision-language models on generated objects with linked questions and known answers, comparing their responses with a human reference. On the color-count task, the best model got every count right on 7.2% of cases, compared with 93.3% for the reference.
Training Qwen3.5-27B further on generated examples raised its score from 3.0% to 68.6% on the task. That improvement shows targeted training can help, while related tests indicate that better counting does not by itself ensure consistent answers from images.
The results come from generated geometric scenes and selected tests of particular models. They do not establish how models perform in complex real-world or embodied settings, or across every model family.
What the evidence shows
- Test complete answers
BedBench is a test of spatial and 3D reasoning in vision-language models. It asks questions about generated geometric objects, including how many blocks of each color they contain.
Its exact-match score counts an answer as correct only when every required part is right, so a plausible answer with one wrong color count still fails. Related questions about the same object also let the test check whether answers agree.
- Measure the full count
The clearest gap appears on color distribution, where every color must be paired with its correct count. The best model scored 7.2% exact match, compared with 93.3% for a human reference based on 30 people answering one randomly drawn question per task.
This is a task-specific comparison, with different item exposure for people and models. Training Qwen3.5-27B further on 800 generated examples raised its score on the task from 3.0% to 68.6%.
- Test the 3D presentation
Researchers changed the visual presentation for Qwen3.5-27B while keeping the 100 objects and counting task: each block appeared as a separate 2D tile instead of in two 3D views. The model’s five-run mean exact-match score rose from 3.0% to 52.4%.
This points to added difficulty in the 3D presentation for this model, but the control changes occlusion, view integration, and layout together, so it cannot isolate the cause.
- Check use of counts
After training further on color counts, Qwen3.5-27B answered a related question about the most common colors correctly on 35.0% of cases from images, versus 74.2% when its own predicted counts were supplied as text. A separate probe recovered exact counts from image features already passed to the language model on 65.0% of benchmark objects, compared with 3.0% for the model’s own zero-shot answers.
These results support difficulty using available information consistently; the probe is not the model’s own answer.
- Check for answer shortcuts
A no-image control shows why accuracy alone cannot establish that a model used visual evidence. On a five-option volume question with the correct answer always in the third position, Gemma4-31B scored 100% without images by always choosing that position, compared with 78.0% with images.
BedBench’s presentation, consistency, and image-removal checks therefore help distinguish getting an answer right from showing how it was reached.
Deep
- Original paper authors
- Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song
- Submitted to arXiv
- License
- CC BY 4.0