LaST-HD: Learning Latent Physical Reasoning from Scalable Human Data for Robot Manipulation

2026-06-22

AI Summary

This work addresses the dynamical mismatch arising from morphological differences between humans and robots by proposing LaST-HD, a method that aligns their physical interaction dynamics in a shared latent space for the first time. The approach constructs a unified latent inference space and trains an action-conditional world model using unpaired human and robot trajectories to generate consistent latent goals, thereby enabling cross-embodiment representation alignment. High-fidelity human demonstrations are collected via a low-cost OOL glove, and a progressive training strategy combining hybrid co-training with online correction is introduced. Remarkably, with only 20 minutes of human demonstration data, the method achieves over 90% task success rates across novel environments, objects, and positions, demonstrating substantially improved generalization capability.

Abstract

Human-hand demonstrations provide a direct and scalable source of physical interaction data for robot learning. While manual retargeting is indispensable for establishing kinematic action correspondence across different morphologies, robust transfer requires going beyond geometry to address the underlying alignment of physical dynamics between human and robot manipulation. To address this, we introduce LaST-HD, a novel human-to-robot action learning paradigm that extends reasoning-before-acting VLA by aligning human-hand and robot demonstrations in a shared latent reasoning space. Rather than mimicking human kinematics, LaST-HD trains an auxiliary action-conditioned world model on unpaired human-hand and robot trajectories to synthesize unified latent targets. After aligning cross-embodiment representations in this shared forward-dynamics space, these targets supervise LaST-HD's latent reasoning process, enabling it to internalize shared physical dynamics and drive efficient human-hand action learning. Moreover, we develop Out-of-Lab (OOL) Glove, a low-cost motion-capture glove tailored to LaST-HD for human-hand data collection. The captured human data provide precise keypoints and serve as universal action supervision across grippers and dexterous hands. Armed with the aligned latent space and high-fidelity human-hand data, we develop a progressive mixed-to-human training recipe comprising mixed human-robot co-training and human-hand online correction post-training. Through mixed co-training, LaST-HD improves generalization to novel objects, scenes, and positions using only human-hand demonstrations. With online correction, LaST-HD further adapts to novel environments and achieves over 90\% accuracy using only 20 minutes of OOL glove data.

Problem

• human-to-robot transfer
• physical dynamics alignment
• robot manipulation
• embodiment mismatch
• action learning from human demonstrations

Innovation

• latent reasoning
• physical dynamics alignment
• human-to-robot transfer
• unpaired trajectory learning
• motion-capture glove