DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation

2025-10-17

AI Summary

This work addresses key challenges in robotic dexterous manipulation—namely, inaccurate contact modeling, poor skill transferability across hand morphologies, and scarcity of real-world interaction data. To this end, we introduce the first large-scale, hybrid real-virtual dataset for human hand manipulation: it encompasses 21 dexterous manipulation tasks and 7,000 hours of high-fidelity multimodal interaction data, synchronized across multi-view RGB-D, high-precision motion capture, and physically validated contact force annotations. Methodologically, we propose a novel “real demonstration → simulation policy” reinforcement learning transfer framework that uniquely integrates human demonstrations, a canonical skill taxonomy, and physically consistent contact modeling. This enables morphology-agnostic skill generalization and fine-grained contact force inference. Experiments demonstrate high-fidelity reproduction of human manipulation behaviors and establish a scalable foundation for contact-rich control and cross-morphology skill transfer.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present DexCanvas, a large-scale hybrid real-synthetic human manipulation dataset containing 7,000 hours of dexterous hand-object interactions seeded from 70 hours of real human demonstrations, organized across 21 fundamental manipulation types based on the Cutkosky taxonomy. Each entry combines synchronized multi-view RGB-D, high-precision mocap with MANO hand parameters, and per-frame contact points with physically consistent force profiles. Our real-to-sim pipeline uses reinforcement learning to train policies that control an actuated MANO hand in physics simulation, reproducing human demonstrations while discovering the underlying contact forces that generate the observed object motion. DexCanvas is the first manipulation dataset to combine large-scale real demonstrations, systematic skill coverage based on established taxonomies, and physics-validated contact annotations. The dataset can facilitate research in robotic manipulation learning, contact-rich control, and skill transfer across different hand morphologies.

Problem

• Bridging human demonstrations with robot learning for dexterous manipulation
• Training policies to reproduce human hand-object interactions in simulation
• Providing physics-validated contact annotations for manipulation research

Innovation

• Combining real human demonstrations with synthetic data
• Training policies via reinforcement learning in simulation
• Providing physics-validated contact annotations and force profiles