WAM-RL: World-Action Model Reinforcement Learning with Reconstruction Rewards and Online Video SFT

2026-06-16

AI Summary

Existing World-Action models rely on expert trajectories, limiting their generalization to out-of-distribution fine-grained manipulation tasks and lacking the capacity for continuous improvement through real-world interaction. This work proposes WAM-RL, a novel framework that introduces reinforcement learning into the World-Action paradigm for the first time, enabling joint online optimization of the world model and action model through environmental interaction and fostering their co-evolution. By integrating hierarchical reinforcement learning, online video-supervised fine-tuning, a reconstruction-based reward mechanism, and a joint training strategy, the method substantially outperforms baselines that optimize only the action model in long-horizon tasks. The results demonstrate that simultaneous online optimization of both components is crucial for enhancing control precision and adaptability.

Abstract

Recent World-Action (WA) models demonstrate strong generalization ability and data efficiency, but they typically rely on expert trajectories for training. This reliance limits their ability to acquire fine-grained manipulation skills beyond the demonstration distribution and prevents them from continuously improving through real-world interaction. To address these limitations, we propose WAM-RL, a reinforcement learning framework that enables joint optimization of the world model and the action model through online interaction with the environment. By allowing the two components to co-evolve, our approach enhances fine-grained control and adaptability. Specifically, a WA model consists of a world model and an actor. We design a tailored reinforcement learning method with hierarchical optimization to coordinate their improvement. On the methodological side, we systematically investigate the effects of applying reinforcement learning to the action model, as well as online training of the world model within an RL setting. Our experiments reveal a key insight: optimizing only the actor yields improvements on short-horizon tasks, but fails to provide significant gains on long-horizon tasks. In contrast, jointly optimizing both the world model and the actor is critical for achieving strong performance in long-horizon settings. Our work is the first to introduce reinforcement learning into the World-Action paradigm, and provides insights into how online optimization of both the action head and the world model impacts overall performance.

Problem

• World-Action models
• expert trajectories
• fine-grained manipulation
• real-world interaction
• long-horizon tasks

Innovation

• World-Action Model
• Reinforcement Learning
• Joint Optimization
• Online Training
• Long-horizon Tasks