MemoryWAM: Efficient World Action Modeling with Persistent Memory

2026-06-18

AI Summary

Existing world action models struggle to balance long-term memory retention and efficient inference in non-Markovian environments, often constrained by short observation horizons or computational costs that scale with sequence length. This work proposes a hybrid memory mechanism that integrates recent frames, event-boundary anchor frames, and a gist-token-based compressed summary of historical context. Coupled with a tailored attention architecture, this approach jointly models short- and long-range dependencies while preserving persistent memory capabilities. The method achieves superior performance over state-of-the-art vision-language-action (VLA) and world action model (WAM) baselines on long-horizon manipulation tasks in both simulated and real-world settings, while substantially reducing inference latency and GPU memory consumption.

Abstract

Robust robotic manipulation in the real world requires not only an understanding of the current observation, but also memory and dynamics modeling. World action models (WAMs) possess these capabilities by jointly modeling visual foresight and actions conditioned on both current and historical observations, making them a promising paradigm for robotic manipulation. However, existing WAMs face a fundamental trade-off: methods with efficient inference typically condition only on a bounded window of recent observations and therefore struggle in non-Markovian environments, whereas methods that preserve long histories incur time and space costs that grow substantially with sequence length. To address this challenge, we introduce MemoryWAM, a world action model with efficient persistent memory. MemoryWAM uses a hybrid memory design that combines recent frames, event-boundary anchor frames, and compact gist tokens that summarize long-range history. A tailored attention mechanism enables retrieval of both detailed short-term context and compressed long-term context, supporting memory-dependent decision-making with reduced inference latency and GPU memory usage. Across long-horizon, memory-dependent manipulation tasks in both simulation and the real world, MemoryWAM outperforms strong vision-language-action (VLA) and WAM baselines while maintaining favorable computational efficiency.

Problem

• World Action Models
• Persistent Memory
• Non-Markovian Environments
• Long-horizon Manipulation
• Efficient Inference

Innovation

• persistent memory
• world action model
• hybrid memory design
• gist tokens
• efficient inference