EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data

2026-07-08

AI Summary

Behavior cloning struggles to disentangle transferable task semantics from human-specific factors—such as embodiment and head motion—when training robots on first-person human demonstration data, leading to limited generalization. To address this, this work proposes the World Action Model (WAM) framework, which learns more transferable world representations by jointly predicting actions and scene dynamics. The study introduces an innovative EgoWAM co-training mechanism that systematically evaluates the impact of different world prediction targets—including pixels, DINO features, and 3D motion flow—on policy transferability, while keeping the policy backbone and action head fixed. Experiments on three real-world bimanual robot tasks demonstrate that DINO features improve out-of-distribution object and scene generalization by up to 4×, while 3D motion flow boosts in-domain performance by 20–30%, both substantially outperforming conventional behavior cloning.

Abstract

Egocentric human data offers scalable supervision for robot manipulation. However, behavior cloning entangles transferable content like objects, scenes, and task semantics, with non-transferable factors like human morphology, head motion, and behavioral style. We study whether World Action Models (WAMs) provide a better training signal by requiring policies to predict not only actions, but also how the scene evolves. The central question is what world representation best enables human-to-robot transfer. We hypothesize that an effective world target should abstract appearance, capture agent-invariant physical effects, and separate camera motion from environment change. We introduce EgoWAM, a controlled human-robot co-training framework that fixes the policy backbone, action head, and data mixture while varying only the world prediction target, comparing Pixel, DINO, and 3D motion flow. Across three real-world bimanual tasks, WAM co-training scales more effectively with in-the-wild egocentric human data than behavior cloning. Pixel-based prediction transfers weakly, while DINO and 3D flow yield substantial gains: DINO improves out-of-distribution object and scene generalization by up to 4x, and 3D flow improves in-domain performance by 20-30%. More details: https://gatech-rl2.github.io/egowam.github.io

Problem

• egocentric data
• human-to-robot transfer
• world representation
• behavior cloning
• generalization

Innovation

• World Action Models
• Egocentric Human Data
• Human-to-Robot Transfer
• DINO Features
• 3D Motion Flow