Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots

2026-06-26

AI Summary

This work addresses the challenge of transferring bimanual manipulation skills from humans to dual-arm robots, which is hindered by noise in human hand pose estimation and mismatches between human hand–object contact patterns and those of parallel-jaw grippers. To bridge this embodiment gap, the authors propose a wrist-relative translation–only action representation that establishes a shared action space between humans and robots. By discarding full 6DoF action signals—particularly rotation components—and integrating a vision–language–action model with interleaved action tokens and attention masking, the method effectively handles missing action information across disparate embodiments. Experiments demonstrate that this approach significantly outperforms 6DoF-based transfer baselines on multiple novel bimanual tasks, with performance consistently improving as the scale of human demonstration data increases.

Abstract

We study whether we can learn novel manipulation skills from human actions to a bi-manual robot with parallel grippers. Human action data is cheap, abundant, and diverse, making it one of the most promising resources for scaling up robot learning. Yet transferring skills from humans to robots remains hard: most prior work treats humans as just another bi-manual 6DoF embodiment, where hand-pose estimates are noisy and the contact patterns of human fingers differ fundamentally from those of a parallel gripper. We argue that learning rotation-inclusive action signals from human data is therefore sub-optimal, and instead propose a bridging action representation: the relative wrist translation within the initial head-camera frame, an action space shared by humans and robots. To handle the potential absence of certain action components in different embodiments, we build a $π_0$-like vision-language-action model with interleaved action tokens and attention masking. On a suite of novel bi-manual manipulation tasks, our bridging action transfers human manipulation knowledge to robots far more effectively than noisy 6DoF human actions and scales with the amount of human data.

Problem

• skill transfer
• human-to-robot
• bimanual manipulation
• action representation
• embodiment gap

Innovation

• bridging action representation
• human-to-robot skill transfer
• vision-language-action model
• bimanual manipulation
• relative wrist translation