MemLearner: Learning to Query Context memory for Video World Models

2026-06-30

AI Summary

Existing video world models struggle to maintain scene consistency over long-term generation due to the absence of an effective memory mechanism, particularly under occlusions and dynamic object interactions. This work proposes MemLearner, which introduces a learnable contextual query mechanism into video world models for the first time. By employing learnable query tokens that adaptively retrieve relevant historical context and directly leveraging the visual priors of pretrained video generative models—without requiring additional trainable modules—MemLearner achieves strong long-term coherence. The method is trained jointly on annotated rendered videos and unlabelled real-world videos across multiple datasets, significantly enhancing scene consistency and memory retention. Empirical results demonstrate clear performance gains over current approaches in complex scenarios involving occlusions and dynamic objects.

Abstract

Video World Models are interactive video generation models that predict future world states based on user actions and history video frames. A critical challenge in video world models is the lack of memory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-based context frame retrieval as memory, but they fail to generalize in scenarios with scene occlusions and dynamic objects. We propose MemLearner, a learning-based adaptive context query method using query tokens to bridge context and predicted tokens. By leveraging the video generation model itself for context querying, MemLearner exploits pre-trained visual priors without training additional modules from scratch, and incorporates efficient strategies for training and inference. We collect a dataset of long videos with scene occlusions and dynamic objects, paired with camera pose annotations, and propose a multi-dataset training strategy leveraging both annotated rendered and unannotated real-world videos. Extensive experiments demonstrate that MemLearner significantly outperforms prior video world models in terms of scene consistency and memory, particularly under challenging occlusion and dynamic scenarios.

Problem

• Video World Models
• memory
• scene consistency
• occlusions
• dynamic objects

Innovation

• MemLearner
• Video World Models
• context memory
• adaptive query
• scene consistency