A coding agent guides video generation with 3D scenes
Video-Coder: Reinforcing Agentic Coding for Video Generation · arXiv:2611.09894
In plain words
A video generator can get object positions, counts, or motion wrong when a prompt describes a precise physical scene.
Video-Coder is a trained coding agent that writes and runs a program to build a 3D scene, checks the rendered draft, and revises the program before a video generator uses the draft as a guide.
In tests of physical consistency and plausibility, videos made with Video-Coder and MiniMax-H3 scored higher on average than videos made directly by MiniMax-H3. The averages cover selected physical categories, not every video.
The research
A prompt can describe exactly where objects should be and how they should move, yet a generated video may not preserve those relationships.
The reported comparisons suggest that making the scene explicit before rendering can improve measured physical consistency for the tested generator pairings, changing what counts as evidence that a video follows a precise prompt.
Video-Coder trains an agent to build a revisable 3D scene draft that guides video generation, improving measured physical consistency in the reported tests.
In the reported tests, Video-Coder's rendered scene draft guides MiniMax-H3, while the comparison system generates directly from the prompt. On PhyGenBench, where GPT-4o judges adherence to prompt-implied physical principles across four categories, the pairing averages 0.67 versus 0.61.
On VBench-2.0, which reports plausibility percentages for three physical dimensions, it averages 78.19% versus 54.58%.
The two averages use different measures, but both favor the guided videos over direct MiniMax-H3 generation in these evaluations.
The scores are averages across specified physical categories or dimensions, so they do not describe every prompt or video.
The material-dimension score remains below CogVideoX-1.5, and the paper attributes that gap to appearance details absent from the draft; it does not test that explanation separately.
What the evidence shows
- Make scene structure explicit
Video-Coder is a trained coding agent for video generation. It turns a text prompt into a Blender program, runs the program to make a 3D scene draft, and revises the code from execution messages and rendered frames.
A downstream video generator then uses the draft's structure and motion to make the final video. This approach addresses a specific problem: direct generation can misplace objects or produce implausible motion when prompts require precise physical relationships.
- Higher physical consistency
PhyGenBench tests whether videos follow physical principles implied by their prompts. GPT-4o scores each video across mechanics, optics, thermal behavior, and material dynamics, with category scores on a 0–1 scale and an average across the four categories.
Video-Coder with MiniMax-H3 scores 0.67 on that average, compared with 0.61 for direct MiniMax-H3 generation. The comparison supports a gain in measured physical consistency for this pairing and test.
- A second measure agrees
VBench-2.0 measures plausibility across three selected physical dimensions using percentages, a different measure from PhyGenBench. The higher average for the guided videos supports the same direction of improvement on this second test, but applies only to those dimensions.
- Training components contribute
The full training recipe scores above the demonstrations-only version and versions that remove either reward in this PhyGenBench test with MiniMax-H3.
This is consistent with the training components contributing in this setup, though the paper does not say whether every version starts from the same checkpoint.
- Transfer has a boundary
Pairing Video-Coder with OmniWeaving provides evidence of transfer to one tested generator. The material-dimension comparison also marks a boundary: the authors suggest that the scene draft lacks appearance details relevant to material behavior, but this explanation was not tested separately.
Deep
- Original paper authors
- Chaoyang Wang, Sicheng Gao, Junan Hu, Kunyang Li, Dongyang Chen
- Submitted to arXiv
- License
- CC BY 4.0