LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models

2026-08-23

AI Summary

该研究提出LD4WAM，通过学习人类视频中的潜在动态来解决世界行动模型中动作不可直接执行的问题，结合语义重建和真实运动对齐，提高机器人在多样化环境下的表现。

Abstract

Human video is playing an increasingly central role in training World Action Models (WAMs), owing to its diversity and low collection cost relative to teleoperated robot data. However, most WAMs learn from such video only by predicting pixel-level future frames, giving dynamics that are not directly actionable, whereas motion retargeting recovers directly actionable actions but leaves a large visual gap across embodiments. We therefore propose motion-aligned latent dynamics as an embodiment-agnostic representation to bridge video priors and low-level actions. We further present LD4WAM, which pairs a Latent Dynamics Model trained with semantic reconstruction and real motion alignment with a World Dynamics Action Model built as a mixture-of-transformers (MoT), which preserves full future-video generation and uses learnable queries to distill these latent dynamics from generated futures for action conditioning. Pretrained on our curated unified dataset of over 5{,}000 hours of human and robot data, LD4WAM performs strongly in RoboTwin simulation and on real robots equipped with both grippers and dexterous hands, while generalizing well to unseen objects and backgrounds.

Problem

• Latent Dynamics
• World Action Models
• Motion Retargeting
• Embodiment-agnostic Representation

Innovation

• latent dynamics
• motion alignment
• mixture-of-transformers (MoT)
• action conditioning