DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning

2024-10-31

arXiv.org

AI Summary

To address the high cost and limited scale of human demonstration data in dexterous bimanual manipulation for humanoid robots, this paper proposes a scalable, automated simulation-based data generation framework. Methodologically, it integrates kinematic retargeting, cross-domain policy distillation, and bimanual coordination modeling within a multi-task Isaac Gym/ROS simulation environment. Leveraging only 60 human demonstrations, the framework autonomously generates 21K high-fidelity, physically consistent bimanual manipulation trajectories. A real-to-sim-to-real闭环 is established to ensure deployment feasibility on physical humanoid platforms. Key contributions include: (i) the first end-to-end automated data synthesis paradigm supporting bimanual dexterous hands; and (ii) substantial improvements in policy generalization and task success rates—demonstrated on both simulated and real-world humanoid robotic sorting tasks. The approach bridges the data scarcity gap while preserving physical plausibility and real-world transferability.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Imitation learning from human demonstrations is an effective means to teach robots manipulation skills. But data acquisition is a major bottleneck in applying this paradigm more broadly, due to the amount of cost and human effort involved. There has been significant interest in imitation learning for bimanual dexterous robots, like humanoids. Unfortunately, data collection is even more challenging here due to the challenges of simultaneously controlling multiple arms and multi-fingered hands. Automated data generation in simulation is a compelling, scalable alternative to fuel this need for data. To this end, we introduce DexMimicGen, a large-scale automated data generation system that synthesizes trajectories from a handful of human demonstrations for humanoid robots with dexterous hands. We present a collection of simulation environments in the setting of bimanual dexterous manipulation, spanning a range of manipulation behaviors and different requirements for coordination among the two arms. We generate 21K demos across these tasks from just 60 source human demos and study the effect of several data generation and policy learning decisions on agent performance. Finally, we present a real-to-sim-to-real pipeline and deploy it on a real-world humanoid can sorting task. Generated datasets, simulation environments and additional results are at https://dexmimicgen.github.io/

Problem

• Automated data generation for bimanual dexterous manipulation
• Reducing human effort in imitation learning for robots
• Simulation-based trajectory synthesis for humanoid robots

Innovation

• Automated data generation via imitation learning
• Simulation environments for bimanual dexterous manipulation
• Real-to-sim-to-real pipeline for humanoid tasks