A new test shows where video-making agents still fall short
VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation · arXiv:2610.35864
In plain words
The paper introduces VibeVideoBench, a test of agents that use code to make complete videos from instructions and optional materials. It checks whether the video is delivered, whether it shows required details, and how judges rate its picture, story, timing, and sound.
Nine agent systems tried the tasks with a one-hour limit. Claude Opus 5.5 had no failed core check and at least one passed on 76.3% of tasks; missing videos counted as failures. The result is specific to these systems, tasks, and time limit.
The research
Can general agents make complete videos whose required details, sound, and timing all work together? Existing tests of agents do not answer this: they check text or functional outputs, while a video must be judged as a perceptual result.
VibeVideoBench tests code-based video agents with task-specific checks and quality judgments, counting delivery as part of whether a task succeeds.
The authors ran nine complete agent systems on the same 239 tasks, with each system using its native harness autonomously under a one-hour limit. Claude Opus 5.5 had the highest strict task success: on 76.3% of tasks, no core check failed and at least one passed; a missing video counted as failure.
The way missing videos are scored also affects comparisons. When they received zero rather than being excluded, Gemini 4 Argon fell from fourth to seventh among the nine systems. This means quality scores for delivered videos alone can hide a system's failure to finish tasks.
These results come from nine complete agent systems running in their native harnesses, autonomously, with a one-hour limit. The comparison does not isolate model effects from harness effects.
A small rerun found that more time helped one system deliver more videos on 20 tasks, and the judges missed injected audio delays, so the findings do not establish performance with longer budgets or reliable judgment of every video.
What the evidence shows
- What the test asks
VibeVideoBench is a test of general multimodal agents that use code to make complete videos from task instructions and optional assets. The question is whether they can produce a finished video whose required content, sound, and timing work together, rather than merely return text or a functional program.
Existing evaluations did not establish this: a video is a perceptual result that ordinary unit tests cannot fully assess.
- How videos are assessed
To make the tasks checkable, the authors rewrote real requests and other source materials into self-contained specifications. These fix the facts, identities, stages, timing, and audio requirements a video needs to meet, while leaving some creative decisions open.
The test then checks delivery with programs, required evidence with task-specific pass-or-fail checks, and quality across five dimensions with three independent judge panels.
- What the main test found
Nine agent systems from six labs attempted all 239 tasks, each running autonomously in its native harness with the same one-hour budget and identical containers. Claude Opus 5.5 had the highest strict task success: on 76.3% of tasks, no core check failed and at least one passed; a missing video counted as failure.
The comparison concerns whole systems, so it does not isolate model performance from harness effects.
- Why delivery matters
Delivery changes how systems compare. When missing videos are scored as zero instead of excluded, Gemini 4 Argon moves from fourth to seventh among the nine systems.
Looking at the 224 runs without a playable video, the authors attributed 169 to final renders that started too late, ran too slowly, or never started. These findings show that a quality score for delivered videos alone does not capture whether a system finished the task.
- What the evidence cannot settle
Two checks qualify the results. Human preferences over playable videos produced system rankings close to the judges' rankings, but the judges missed injected audio delays, so the agreement does not establish reliable judgment of every video.
In a separate 20-task control, GLM-5.3-Flash delivered more videos with longer time limits. That result applies to this system and small task set, not to all agents or tasks.