WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos

2026-07-13

AI Summary

This work addresses the scalability challenge of robot learning from demonstrations, which typically rely on costly action annotations. To overcome this limitation, the authors propose WALA, a framework that jointly leverages labeled robotic demonstration data and unlabeled human–robot interaction videos to learn executable latent action representations deployable on real robots. The approach pretrains a semantics- and geometry-aware latent action model and integrates multi-task joint supervision during policy training through action prediction, latent goal matching, and future state prediction based on DINOv3 features and depth-aware spatial representations. This study presents the first end-to-end method for learning executable latent actions from mixed data sources, achieving a new state-of-the-art average success rate of 75.2% on RoboCasa and demonstrating significantly improved generalization to real-world tasks.

Abstract

Generalizable robot policies typically rely on action-labeled robot demonstrations, which are expensive to collect and difficult to scale. In contrast, large-scale human and robot videos contain rich physical interactions but often lack executable robot action labels. We present WALA, a framework for learning executable latent actions from both action-labeled demonstrations and action-free videos. WALA first pretrains a semantic-geometric latent action model from videos by modeling the evolution between current observations and sparsely sampled future observations. Instead of reconstructing raw pixels, WALA predicts future deltas in the DINOv3 feature space and dense depth space, preserving task-relevant semantic and geometric structure while reducing sensitivity to appearance details. During policy training, the pretrained encoder provides stable latent action targets, and the decoder serves as a trainable latent world model. The latent actions generated by the vision-language backbone are jointly supervised by robot action prediction, latent action target matching, and future dynamics prediction. This enables action-labeled demonstrations to provide executable control supervision, while action-free videos contribute dynamics supervision without requiring robot action annotations. Experiments show that WALA achieves strong performance on RoboTwin, sets a new state-of-the-art result on RoboCasa with 75.2% average success, and improves both policy performance and generalization in real-world manipulation tasks.

Problem

• executable actions
• action-labeled demonstrations
• action-free videos
• robot policy generalization
• latent actions

Innovation

• latent actions
• action-free videos
• semantic-geometric modeling
• DINOv3 features
• world model