In plain words
A visual assistant may give advice while someone is cooking, doing housework, or using a phone. Tests based on recorded conversations cannot show whether a person changes what they do after hearing that advice.
Interactive Visual Gym puts a simulated user in a working task simulation. The user acts, gets advice, and can change course. In these simulations, the best model for completing tasks finished 53.4% within the time limit on average across the three task environments.
The research
Advice given during a task can change what happens next, so fixed transcripts leave a key part of assistance untested. A way to observe that back-and-forth could change what counts as convincing evidence for a visual assistant: success must include the evolving task, not just a plausible reply.
Interactive Visual Gym is a framework that tests and trains visual assistants by letting simulated users act in executable task worlds and respond to their guidance.
Researchers ran assistants through 315 simulated cooking, household, and phone scenarios, where users could change actions after guidance. IVG-Bench measures the share of tasks whose engine-checked goal is reached within its time limit, and judges replies against the simulated world state.
Gemini 4 Argon, the strongest task-completion model, finished 53.4% of tasks on average across the three environments.
In CookSim, measured response delays instead of fixed two-tick timing lowered its on-time completion from 60.9% to 33.1%, showing that reply timing can change outcomes in these simulations.
The tasks and users were simulated. The latency comparison used newly generated interactions, and the training result came from fixed CookSim validation scenarios; the authors caution that simulator performance is not evidence of real-world safety.
What the evidence shows
- Advice changes the task
Interactive Visual Gym is a framework for evaluating and training visual assistants by joining executable task environments with simulated users. The problem it addresses is that replay tests keep recorded user actions fixed, even when an assistant offers different guidance. They therefore cannot show whether advice changes the next action or affects task completion.
- Completion leaves a gap
IVG-Bench tests assistants as they guide simulated users through cooking, household, and phone-interface scenarios. It measures the share of tasks whose simulator-checked goal is reached within the time limit, and scores replies against the simulated world state.
Gemini 4 Argon led task completion, but finished only 53.4% of tasks within the limit on average across the three environments.
- Timing changes outcomes
The latency comparison shows why response timing matters to that completion measure. In CookSim, Gemini 4 Argon’s share of episodes completed within the time budget fell from 60.9% with fixed two-tick timing to 33.1% when measured response delays were used.
This contrast shows that a result under fixed timing can overstate completion under the tested real response delays.
- Scores meet human preferences
The researchers also checked whether their reply-quality measure tracked human preferences. In a CookSim browser study, its scores identified the participant-preferred assistant in 78.5% of eligible participant–assistant-pair comparisons, with ties in metric scores receiving half credit.
Yet simulated users spoke and replied far more often than participants, so preference alignment does not erase the behavioral mismatch.
- Training shows a signal
A CookSim training demonstration suggests the simulations can provide a learning signal: after 12 reward-based updates, Qwen3.5-4B’s mean on-time task completion rose from 33.3% to 42.2% on the tested validation scenarios.
Error-detection recall declined in the same comparison, and the validation scenarios shared recipe and layout cells with training. The result supports a proof of concept, not broad transfer.
Deep
- Original paper authors
- Chenhao Zheng, Zixian Ma, Kai Li, Kanchana Ranasinghe, Jieyu Zhang
- Submitted to arXiv
- License
- CC BY 4.0