BirdDreams uses top-down video to model driving motion
BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model · arXiv:2611.14815
In plain words
A driving world model creates videos of what might happen next. BirdDreams makes two matching videos: one from the driver’s view and one like a simple map from above, so vehicle movement and road layout are easier to track. It can also use a requested path for the driver’s car while predicting how other cars move.
In tests where the driver’s path differed from the recorded drive, people more often preferred BirdDreams’ surrounding-car behavior to a comparison model. That evidence comes from selected driving examples and human judgments, not every road situation.
The research
A simulated drive needs other cars to respond when the driver takes a different path, and the generated driver’s motion should follow that path.
BirdDreams’ results shift the test from replaying recorded scenes toward checking both path following and perceived vehicle reactions under changed actions.
BirdDreams jointly generates camera-view and top-down driving videos, using explicit road-relative motion to guide requested ego paths and predict surrounding-vehicle behavior.
For trajectory control, the researchers supplied the models with the real ego-vehicle path, then compared generated positions with positions reconstructed from real videos.
BirdDreams had lower average displacement error than OmniDreams (Plücker) on both driving datasets, indicating closer path following in that evaluation.
A separate counterfactual comparison changed the ego action while holding initial views and road geometry the same.
Human preferences and selected examples favored BirdDreams’ surrounding-vehicle behavior, supporting the idea that its explicit scene layout helps model reactions as well as the ego path.
The trajectory results come from benchmark scenes with ground-truth ego paths supplied. Evidence about surrounding vehicles comes from selected examples and human judgments under tested counterfactual maneuvers with road geometry supplied; it does not cover every driving scene or maneuver.
What the evidence shows
- Make movement explicit
BirdDreams is a driving world model that jointly generates perspective video and a schematic top-down view of the same scene. The top-down view makes road-relative vehicle positions and motion explicit, while the perspective video retains visual details.
This addresses a central difficulty: in camera video alone, appearance, distance, and obstruction can make vehicle movement hard to learn.
- Guide the driver’s path
The model also uses its predicted top-down view to follow a requested path for the ego vehicle, meaning the vehicle whose camera view is being generated. It shifts and rotates the view along that path, then uses the result as a soft guide while generating the future.
Other vehicles remain predictions, so they can move in response rather than simply replaying recorded paths.
- Compare vehicle reactions
With the same initial views, changed ego actions, and static road geometry, participants more often preferred BirdDreams’ surrounding-vehicle behavior to OmniDreams’ (57% versus 24% of 375 judgments).
This measures perceived realism under those tested conditions, with each model predicting surrounding-vehicle motion.
- Check path accuracy
A separate test asks whether the generated ego vehicle follows a supplied path. Average displacement error measures the average distance between generated and real-video ego positions at matching times.
With the real trajectory supplied, BirdDreams had 35% lower error than OmniDreams (Plücker) on PhysicalAI-AV and 36% lower on nuScenes. These results support trajectory control on the evaluated driving datasets, alongside the human evidence about other vehicles.
Deep
- Original paper authors
- Junyoung Seo, Jaewoo Seo, Seungryong Kim, Laura Leal-Taixé, Xiuming Zhang
- Submitted to arXiv
- License
- CC BY 4.0