Compression and Retrieval: Implicit Memory Retrieval for Video World Models

2026-06-22

AI Summary

This work addresses the challenge of maintaining long-term memory consistency in video world models under complex camera trajectories. To this end, we propose an attention-based implicit memory retrieval mechanism that enables flexible memory access through viewpoint-aware positional encoding, coupled with a lightweight context compression network for efficient long-sequence processing. Our key innovation lies in the first integration of viewpoint information into positional encoding and its synergistic combination with attention mechanisms for implicit memory modeling. To facilitate training and evaluation of long-horizon video world models, we introduce SceneFly, a large-scale synthetic dataset. Experiments demonstrate that our approach achieves state-of-the-art performance across multiple benchmarks and exhibits strong generalization capabilities in open-domain scenarios.

Abstract

Video world models hold promise for simulating interactive environments, yet maintaining consistent long-term memory across complex camera trajectories remains a critical challenge. Existing methods typically rely on computationally expensive context scaling or rigid heuristic retrieval mechanisms, which lacks generalization to varying camera trajectories and environments. In this paper, we propose Compression and Retrieval (CaR), an attention-driven implicit memory retrieval mechanism to overcome these limitations. By injecting viewpoint information via positional encoding, our method performs flexible memory retrieval through attention computation. To efficiently process extended contexts with minimal computational overhead, we further introduce a lightweight context compression network. Furthermore, we construct SceneFly, a large-scale synthetic dataset featuring realistic camera trajectories and frame-level annotations to train and evaluate long-horizon video world models. Extensive experiments demonstrate that our approach achieves state-of-the-art results on established benchmarks and exhibits strong generalization to open-domain scenes.

Problem

• video world models
• long-term memory
• camera trajectories
• memory retrieval
• generalization

Innovation

• implicit memory retrieval
• attention mechanism
• context compression
• video world models
• camera trajectory generalization