ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

2026-06-16

AI Summary

This work addresses the limitation of existing interactive world models, which are largely confined to navigation actions and struggle to support fine-grained object interactions such as grasping or opening doors, resulting in environments that are visually plausible but functionally inert. To overcome this, the authors propose ActWorld, a unified framework that jointly models navigation and object interaction within a chunked autoregressive architecture. By introducing a large-scale dataset of 100,000+ interaction videos annotated with chain-of-thought reasoning, along with an action-aware hierarchical memory mechanism and persistent object identity tokens, ActWorld effectively mitigates action forgetting. The model achieves, for the first time, coherent joint generation of navigation and detailed object interactions in general scenes, significantly enhancing interaction realism and temporal consistency while maintaining precise viewpoint control, thereby outperforming baseline models limited to navigation-only capabilities.

Abstract

Interactive world models aim to simulate environment dynamics under real-time user actions. However, their action vocabulary is largely confined to navigation: most actions correspond to motion (e.g., walk, turn, look around), while interaction with objects in the scene (e.g., pick up plates, open doors, or trigger physical responses) is either absent, restricted to game domains, or relegated to prompt-to-full-video scenarios. The resulting worlds are visually explorable but not truly actionable. In this work, we present ActWorld, an interactive world model that extends prior navigation-centric generators to support mid-rollout object interaction within a chunk-autoregressive framework. We argue that the navigation-interaction gap stems from two bottlenecks. First, a data bottleneck: the lack of human-object interaction data with accurate, dense labels. Second, a memory bottleneck: recency-biased history compression in existing world models discards the event-transition frames that causally determine subsequent object states, leading to an action-forgetting pathology. On the data side, we construct a 100K interaction video dataset, each annotated with per-chunk captions via chain-of-thought reasoning. On the model side, we introduce a hierarchical action-aware memory design that routes history compression by interaction importance, complemented by a persistent memory bank that maintains event-update and object-identity tokens across long rollouts. Experiments show that ActWorld supports both flexible navigation and rich object interaction within a single model, substantially improving interaction fidelity over navigation-only baselines without sacrificing viewpoint control. Project page is available at https://interactwm.github.io/ActWorld.

Problem

• interactive world models
• object interaction
• action vocabulary
• navigation-interaction gap
• action-forgetting

Innovation

• interactive world model
• action-aware memory
• object interaction
• chunk-autoregressive generation
• persistent memory bank