Unified Motion-Action Modeling for Heterogeneous Robot Learning

2026-06-15

AI Summary

This work addresses the challenge of unifying heterogeneous robotic data—such as human videos, simulated demonstrations, and real-world trajectories—for multitask learning and flexible inference without relying on manual task instructions. The authors propose the Unified Motion–Action (UMA) model, which leverages 3D object motion trajectories as a shared interface to jointly model visual motion control and dynamics for the first time. By employing mask-based generative learning, UMA dynamically switches between supervised training and inference modes, while integrating hindsight relabeling and contrastive learning to disentangle task intent from scene geometry. The method enables end-to-end pretraining across diverse data sources and surpasses specialized state-of-the-art approaches in few-shot task transfer, vision-based control under motion constraints, and dynamics modeling.

Abstract

We present Unified Motion-Action (UMA) Model, an approach that uses 3D object motion trajectories as a shared interface to bridge visuomotor control and dynamics modeling. UMA treats object motion and robot actions as co-evolving variables under a masked generative objective, in which the mask pattern determines both the supervision regime during pretraining and the inference mode at deployment. Using hindsight-relabeled motion contexts and a contrastive objective that disentangles task intent from scene geometry, UMA enables multi-task pretraining across heterogeneous data sources without requiring manually annotated task instructions. At deployment, the same pretrained parameters support motion-conditioned visuomotor control, motion-based dynamics modeling, and task adaptation from few-shot demonstrations. Pretrained on a mixture of robot demonstrations, human videos, and simulated data, UMA consistently outperforms state-of-the-art baselines specialized for each inference mode.

Problem

• heterogeneous robot learning
• visuomotor control
• dynamics modeling
• multi-task pretraining
• object motion trajectories

Innovation

• Unified Motion-Action Modeling
• 3D motion trajectories
• masked generative modeling
• heterogeneous robot learning
• contrastive disentanglement