A simulator tests assistants as advice changes events
InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants · arXiv:2610.33966
In plain words
A helpful assistant may need to change its advice as a person acts and sees new things. This paper introduces a simulator where an assistant gives spoken guidance to a simulated person, who acts and then gets a new view of the scene. A companion test set checks whether tasks are actually completed in the simulator.
Across 12 tested assistant models, the top model completed 41.7% of 120 tasks; each other model completed 33.3% or fewer. These are simulator results with a simulated person carrying out the instructions.
The research
Can an assistant still help when its advice changes what a person does and sees, turn by turn? Fixed-video tests cannot answer that because the footage stays the same after a response, while studies with real people are costly to scale. The paper builds a simulated way to test this changing interaction.
InteractiveWorld simulates advice-driven human action, and InteractiveBench measures whether assistants can complete multi-turn tasks in that changing world.
InteractiveBench is a test of whether an assistant can guide a simulated person when each instruction changes what happens next. The evaluator checked task completion from the simulator’s final state, with Gemini 3.1 Pro acting as the Human agent.
Claude Fable 5.1 completed 41.7% of tasks, the highest average among 12 models; each other model completed 33.3% or fewer.
For Fable 5.1, the rate was 85% on Decision tasks but 20% on Guidance and 25% on Planning, with 20 tasks in each family. This gap shows why an overall score alone does not capture performance across task types.
These results describe simulated episodes with Gemini 3.1 Pro acting as the Human agent. A preliminary comparison with human participants inside the simulator does not establish equivalent performance or show how assistants perform in physical-world use.
What the evidence shows
- Testing help that changes events
InteractiveBench is a test set for multimodal language models acting as first-person-view assistants in a simulated environment. It tests whether an assistant can guide a simulated person through changing, multi-turn tasks; a pass means all required objectives were completed, as checked from the simulator’s final state.
This addresses a limitation of fixed-video tests, where an assistant’s answer cannot change what the person sees or does next.
- Average scores hide task gaps
The evaluator tested 12 assistant models with Gemini 3.1 Pro fixed as the simulated Human agent. Claude Fable 5.1 had the highest average pass rate, completing 41.7% of its 120 tasks; each of the other models completed 33.3% or fewer.
The average was not uniform across task types: Fable 5.1 passed 85% of Decision tasks, but 20% of Guidance and 25% of Planning tasks, with 20 tasks in each family.
- Tracing the first failure
To understand what went wrong, researchers reviewed valid failed episodes and labeled the earliest mistake that meaningfully caused each failure. Across models, visual-perception and memory errors made up 70.4% to 85.7% of first errors among model-attributable failed episodes.
This denominator excludes successful tasks and infrastructure or simulator failures, so the range describes the mix of failures, not the share of all tasks that failed for these reasons.
- Checking task design and memory
Two checks help interpret the task results. Under the reported protocol, the same simulated Human agent passed 100% of tasks with an Oracle Assistant and 0% without assistance, showing that the tasks could be completed with guidance and were not completed unaided by that agent.
In a separate comparison, adding an accumulating memory entry improved pass rates by 5 to 15 percentage points for three models on Prevention and Finding tasks, but effects elsewhere were mixed.
- Where the evidence stops
The results describe a simulator in which a model-driven Human agent follows assistant guidance. The authors also report a preliminary comparison with human participants acting as executors inside the simulator, but it used the same tasks and was not a statistical equivalence test.
The scores therefore characterize the tested simulated interactions; they do not establish how assistants perform with real people in physical settings.
Deep
- Original paper authors
- Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu
- Submitted to arXiv
- License
- arXiv non-exclusive