DIM-WAM: World-Action Modeling with Diverse Historical Event Memory

2026-06-25

AI Summary

This work addresses the limitations of existing world-action models, which struggle with long-horizon manipulation tasks that depend on early observations and task progression due to their reliance on short-term history and near-future predictions. To overcome this, the authors propose a memory-augmented world-action model that jointly captures multi-scale historical context, local future dynamics, and global task progress to enable event-memory-guided video and action denoising. Key innovations include a similarity-based multi-memory bank mechanism, a task-progress supervision objective, and a memory read-write strategy integrating visual event extraction with identity and temporal embeddings. Experiments demonstrate substantial performance gains, improving success rates on RMBench from 28.4% to 69.8%, and achieving 91.5% and 80.0% success rates on stage-wise and full-task evaluations, respectively, in real-world Franka robot experiments.

Abstract

World-action models have shown promising robot-manipulation performance by jointly predicting future visual states and actions. However, existing methods mainly rely on short-term history and short-horizon future prediction, which is insufficient for long-horizon tasks whose correct execution depends on earlier observations and task progress. Such temporally dependent tasks require effective use of complementary temporal information, including recent local context, cross-stage historical events, immediate future dynamics, and global task progress. To address long-term forgetting and poor awareness of the global task state, we introduce DiM-WAM, a memory-augmented world-action model that integrates multi-scale historical context, local future dynamics, and global task progress. The memory extracts compact visual event information from real observations, updates multiple memory banks through independent similarity-based merging, and then reads the bank-identity- and time-embedded long-term context to condition video and action denoising. A progress-supervision objective further encourages memory tokens to encode not only completed historical events but also the current task stage and its implications for the remaining task. On RMBench, DiM-WAM raises average success from 28.4% with LingBot-VA to 69.8%, exceeding the explicit-memory Mem-0 baseline at 42.0%. On four real-world Franka tasks, it improves average stage success from 70.7% to 91.5% and full-task success from 52.5% to 80.0%. Project page: https://wangkai-casia.github.io/dim-wam/{\texttt{https://wangkai-casia.github.io/dim-wam/}}.

Problem

• world-action modeling
• long-horizon tasks
• temporal dependency
• historical event memory
• task progress awareness

Innovation

• memory-augmented world-action model
• multi-scale historical context
• global task progress
• similarity-based memory merging
• progress-supervised denoising