WORLDMEM: Long-term Consistent World Simulation with Memory

2025-04-16

AI Summary

World simulation suffers from deteriorating long-term spatiotemporal consistency—particularly in 3D scene evolution—due to the limited context window of Transformer architectures. To address this, we propose the first world simulation framework enabling long-term consistency. Our method introduces: (1) a stateful memory bank jointly indexed by pose and timestamp; (2) a state-aware memory attention mechanism that enables precise cross-view and temporal scene reconstruction; and (3) a joint spatiotemporal representation coupled with an end-to-end differentiable reconstruction module. Evaluated on both synthetic and real-world scenes, our approach reduces reconstruction error by 42% under large viewpoint shifts and extended time horizons, significantly improving long-term consistency metrics. Moreover, it supports continual perception and physics-aware interaction.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing a memory attention mechanism that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.

Problem

• Maintains long-term 3D spatial consistency in world simulation
• Reconstructs scenes accurately across viewpoint or time gaps
• Models dynamic world evolution for perception and interaction

Innovation

• Memory bank stores frames and states
• Memory attention extracts relevant information
• Timestamps model dynamic world evolution