VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory

2025-06-23

AI Summary

This work addresses two key challenges in interactive video generation: cumulative view extrapolation errors and long-range scene inconsistency. To this end, we propose Surfel-Indexed View Memory (VMem), a geometrically grounded view memory mechanism that indexes historical observations using 3D surfels (surface elements). VMem enables structured storage and retrieval of visual history anchored to surfel geometry, eliminating reliance on short context windows. By integrating selective context attention with lightweight 3D surface reconstruction, VMem achieves efficient and geometrically consistent long-horizon video synthesis. On long-duration scene benchmarks, our method significantly improves scene coherence and camera controllability while reducing computational overhead. Crucially, it breaks the conventional trade-off between context length and reconstruction error inherent in prior approaches, enabling scalable, high-fidelity interactive video generation.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multi-instance/Multi-view Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We propose a novel memory mechanism to build video generators that can explore environments interactively. Similar results have previously been achieved by out-painting 2D views of the scene while incrementally reconstructing its 3D geometry, which quickly accumulates errors, or by video generators with a short context window, which struggle to maintain scene coherence over the long term. To address these limitations, we introduce Surfel-Indexed View Memory (VMem), a mechanism that remembers past views by indexing them geometrically based on the 3D surface elements (surfels) they have observed. VMem enables the efficient retrieval of the most relevant past views when generating new ones. By focusing only on these relevant views, our method produces consistent explorations of imagined environments at a fraction of the computational cost of using all past views as context. We evaluate our approach on challenging long-term scene synthesis benchmarks and demonstrate superior performance compared to existing methods in maintaining scene coherence and camera control.

Problem

• Maintaining long-term scene coherence in video generation
• Reducing computational cost in interactive video synthesis
• Improving camera control for consistent environment exploration

Innovation

• Surfel-Indexed View Memory for past views
• Efficient retrieval of relevant past views
• Consistent scene synthesis with reduced cost