Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation

2026-06-17

AI Summary

This work addresses the limitations of existing action-conditioned world models in continuous robotic manipulation, which are prone to detail loss or hallucination due to end-effector occlusions and rapid wrist-mounted camera motion. To overcome these challenges, the authors propose W-VMem—a memory-augmented, multi-view action-conditioned world model that uniquely integrates surfel-based scene evolution with action-conditioned prediction. W-VMem constructs a wrist-centric 4D memory structure that explicitly stores and geometrically aware retrieves historical observations. Leveraging action-guided memory rendering, multi-view consistency constraints, and a scoring mechanism, the model generates temporally persistent and coherent rollouts in complex manipulation tasks. Experiments demonstrate a 14.5% improvement in Pearson correlation between policy evaluation and real-world performance over Ctrl-World, and an increase in long-horizon task success rate from 58% to 72%.

Abstract

Action-conditioned world models have emerged as a promising paradigm for robot learning, offering a scalable alternative to costly real-world experimentation by generating action-consistent video rollouts. However, persistent world modeling remains challenging in manipulation: frequent end-effector occlusions and rapid wrist-camera motion make the current observation insufficient for predicting future views, causing models to forget or hallucinate scene details seen in earlier frames. Existing memory retrieval strategies often fail to identify informative history in dynamic manipulation scenarios. To address this limitation, we propose Mem-World, a memory-augmented multi-view action-conditioned world model. At its core, we present W-VMem, a 4D wrist-view-centered surfel-indexed memory that anchors historical observations to temporally evolving surface elements. By explicitly modeling when and where scene elements are observed, W-VMem enables geometry-aware retrieval of relevant history frames conditioned on future actions. During generation, relevant history frames are selected via surfel-based rendering and scoring, providing informative and non-redundant context for prediction. Extensive experiments show that Mem-World generates persistent rollouts in complex manipulation scenarios, enables more reliable policy evaluation than Ctrl-World, improving the Pearson correlation with real-world performance by 14.5\%, and supports effective policy improvement through synthetic data generation, increasing success rates from 58\% to 72\% on long-horizon tasks.

Problem

• persistent world modeling
• action-conditioned world models
• occlusion
• memory retrieval
• robot manipulation

Innovation

• memory-augmented world models
• surfel-indexed memory
• action-conditioned video prediction
• persistent scene modeling
• geometry-aware retrieval