Do as I Do: Dexterous Manipulation Data from Everyday Human Videos

2026-06-17

AI Summary

This work addresses the key challenge in embodied intelligence of efficiently generating manipulation data for dexterous multi-fingered robotic hands from everyday monocular RGB videos. The authors propose DO AS I DO, a novel algorithm that, for the first time, integrates in-the-wild multi-view videos to reconstruct and transfer human manipulation skills to real robots through vision-driven hand–object pose estimation, physics-constrained optimization, and cross-embodiment motion retargeting. This approach substantially enhances the feasibility of producing high-fidelity, executable manipulation trajectories directly from ordinary videos, outperforming existing methods on both real-world datasets and internet-sourced footage. The paper also provides practical guidelines for practitioners on collecting effective human demonstration videos.

Abstract

How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-robot embodiment gap have hindered the adoption of abundant monocular RGB-only human videos as the primary source of robot manipulation data. In this work, we present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various egocentric and exocentric in-the-wild video sources. The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding robot-complete manipulation data from disparate human videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos, as we show in experiments on datasets with ground truths and on a dataset of video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation.

Problem

• dexterous manipulation
• human videos
• hand-object interaction
• embodiment gap
• robotic data generation

Innovation

• dexterous manipulation
• hand-object interaction
• video retargeting
• embodiment gap
• monocular RGB video