H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos

2025-05-17

AI Summary

This work addresses the significant morphological gap hindering the transfer of human-hand video pre-trained models to robotic visual representation learning. To bridge this gap, we propose the first end-to-end “human → robot” video data augmentation paradigm: leveraging human hand keypoint detection, robotic kinematic simulation, and neural rendering to embed synthesized robot motions into egocentric human videos—yielding a million-scale, robot-centric video dataset covering UR5, Franka, and LeaPhand platforms. We further introduce a CLIP-driven semantic fidelity metric for quantitative evaluation. On simulation benchmarks (RoboMimic, RLBench, PushT) and real-world UR5 manipulation tasks, downstream policies achieve +5.0–10.2% (simulation) and +6.7–23.3% (real-world) improvements in success rate, demonstrating substantially enhanced cross-domain generalization.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Vision for Robotics & Autonomous Driving

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large-scale pre-training using videos has proven effective for robot learning. However, the models pre-trained on such data can be suboptimal for robot learning due to the significant visual gap between human hands and those of different robots. To remedy this, we propose H2R, a simple data augmentation technique that detects human hand keypoints, synthesizes robot motions in simulation, and composites rendered robots into egocentric videos. This process explicitly bridges the visual gap between human and robot embodiments during pre-training. We apply H2R to augment large-scale egocentric human video datasets such as Ego4D and SSv2, replacing human hands with simulated robotic arms to generate robot-centric training data. Based on this, we construct and release a family of 1M-scale datasets covering multiple robot embodiments (UR5 with gripper/Leaphand, Franka) and data sources (SSv2, Ego4D). To verify the effectiveness of the augmentation pipeline, we introduce a CLIP-based image-text similarity metric that quantitatively evaluates the semantic fidelity of robot-rendered frames to the original human actions. We validate H2R across three simulation benchmarks: Robomimic, RLBench and PushT and real-world manipulation tasks with a UR5 robot equipped with Gripper and Leaphand end-effectors. H2R consistently improves downstream success rates, yielding gains of 5.0%-10.2% in simulation and 6.7%-23.3% in real-world tasks across various visual encoders and policy learning methods. These results indicate that H2R improves the generalization ability of robotic policies by mitigating the visual discrepancies between human and robot domains.

Problem

• Bridges visual gap between human and robot hands
• Generates robot-centric data from human videos
• Improves robot learning success rates significantly

Innovation

• Detects human hand keypoints for robot synthesis
• Composites rendered robots into egocentric videos
• Generates robot-centric data from human videos