Explicit view controls help video models separate motion and switch views
World-in-Context: Interactive Video World Modeling with In-Context Conditioning · arXiv:2610.33169
In plain words
World-in-Context is a framework for generating video that responds to controls. It gives the model explicit signals for the view, character actions, and camera movement, so the model can consider them together while creating what happens next.
In 100 navigation scenarios, explicit view signals produced different motion patterns for first-person and free-camera views, while text descriptions alone produced similar patterns. This is evidence about those scenarios, not every kind of interactive video.
The research
When an interactive video model receives a camera path or movement command, how does it know whether the scene should move like a first-person view or a free-moving camera?
The paper asks whether making the view explicit can help coordinate these controls and support switching perspectives in generated video.
World-in-Context is an interactive video framework that puts view, action, and camera controls together so a model can coordinate them when generating future video.
World-in-Context puts view, action, and camera controls together when generating video. Does making the view explicit help the model produce motion that fits the perspective and switch views on request?
In 100 scenarios with descriptions and forward actions fixed, explicit view tokens separated first-person and free-camera motion profiles that text-only descriptions left similar.
On WBench’s first-person to third-person test, transition success was 93.8% after training that included switching sequences, versus 37.5% after single-view training.
These results concern the reported framework, training data, and evaluation settings. The motion diagnostic covers 100 navigation scenarios, and the switching results cover WBench’s first-person to third-person subset; they do not establish performance across other tasks or settings.
What the evidence shows
- The control coordination problem
World-in-Context is an interactive video world modeling framework: it generates future video in response to a viewing perspective, character actions, and camera motion. The paper addresses a coordination problem.
View modes can be left implicit in descriptions, and separate control pathways can make it harder to represent how view, action, and camera inputs interact.
- A shared representation
The framework represents view, action, and camera controls as tokens alongside the video’s internal representations, where shared attention can relate them. It distinguishes free-camera, first-person, and third-person views explicitly.
Its temporal attention encoding also tracks the ordered control changes between frames, rather than representing only the endpoints.
- Testing view-dependent motion
To check whether explicit view signals correspond to distinct motion, researchers fixed the world description and forward action in 100 navigation scenarios. They reconstructed generated camera paths and measured sideways and vertical oscillation relative to forward travel.
Text-only view descriptions produced closely overlapping first-person and free-camera profiles; explicit view tokens produced more periodic first-person motion and smoother free-camera motion.
- What the evaluations show
WBench tests interactive video navigation and requested viewpoint transitions. On its first-person to third-person switching subset, World-in-Context met the transition criteria on 37.5% of turns after single-view training and 93.8% after a second stage that included switching sequences.
On WBench Navigation, its average score was 81.5, close to 81.6 for the larger EchoWM model. These results support performance in the reported tests, with their specific training and evaluation scopes.
Deep
- Original paper authors
- Harold Haodong Chen, Rui Peng, Rongjin Guo, Lingteng Qiu, Kejie Qiu
- Submitted to arXiv
- License
- CC BY 4.0