A 3D scene memory helps video worlds stay consistent on return
L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory · arXiv:2610.40614
In plain words
L3G-World is a system that makes video worlds from an image and a camera route. It keeps a 3D scene memory: it adds details in places the memory does not cover well, revises details that no longer match, and uses the memory to guide each new video chunk.
On two test datasets, its frames matched most closely among the evaluated methods when the camera returned to the same views. The authors note that the system is mainly designed for static scenes.
The research
When a generated video world is explored for a long time, will it still show a familiar place consistently when the camera returns? Video-history methods can grow costly as more frames accumulate, while 3D scene memories can be slow to reconstruct or render. The research asks how to keep revisited places recognizable while generation remains fast.
L3G-World alternates a revisable 3D scene memory with video generation, rendering stored scene features directly into the generator to support fast, spatially consistent exploration.
Researchers tested L3G-World and other methods on the test splits of DL3DV and RealEstate10K. Each 481-frame camera route moved away from its starting view and then retraced the same path, letting researchers compare generated frames at matching poses.
L3G-World ranked best among the evaluated methods on two measures of frame agreement in both datasets. This supports the conclusion that its persistent memory can help preserve scene content when the camera revisits a place.
The results come from the test splits of two datasets and comparisons among the methods evaluated; they do not show that every generated view stays consistent.
The authors also identify a static-scene focus, possible memory growth as explored areas become very large, and error propagation during long generation.
What the evidence shows
- The long exploration problem
L3G-World is a real-time world-generation system that makes video from an input image and a camera route. The challenge is to keep places recognizable when the camera returns after a long exploration, while avoiding memory and rendering costs that grow with the video stream.
Earlier approaches use video history or explicit 3D scene memory, each with scaling or speed trade-offs.
- A memory that gets revised
The system alternates between a memory updater and a video generator. The updater adds 3D Gaussians, compact elements representing the scene, where the memory lacks coverage or reproduces the scene poorly.
It also revises visible existing elements using differences from the latest observation, rather than allowing all earlier scene details to remain fixed.
- Direct guidance for new video
Each scene element stores a feature in the video generator's compressed representation. L3G-World renders those features for the next camera view and uses them to guide video generation directly, avoiding the extra step of rendering an RGB image and encoding it again.
The generated chunk then informs another memory update, extending the same persistent scene representation.
- Frames agree on return
On test splits of DL3DV and RealEstate10K, researchers tested routes that moved away from a starting view and then retraced the path. They compared frames generated at matching camera poses; L3G-World ranked best on two measures of frame agreement in both datasets among the evaluated methods.
The result supports revisitation consistency in these tests, not a claim about every generated view.
- Refinement matters in a control
In a DL3DV ablation, disabling memory refinement worsened several measures, including self-consistency, camera accuracy, and 3D consistency, compared with the full system. Without refinement, newly added scene elements stayed fixed as the memory accumulated.
The authors interpret the result as evidence that revising the memory helps correct accumulated reconstruction errors, though this control does not establish the effect across all settings.
Deep
- Original paper authors
- Jinhyeok Choi, JoungBin Lee, Jongmin Lee, Minkyeong Jeon, Jiyoung Kim
- Submitted to arXiv
- License
- CC BY 4.0