HRDexDB: A Large-Scale Dataset of Dexterous Human and Robotic Hand Grasps

2026-04-16

AI Summary

This work addresses a critical limitation in dexterous manipulation research—the absence of large-scale, multimodal datasets that simultaneously capture high-fidelity human and diverse robotic grasping behaviors. To bridge this gap, the authors introduce HRDexDB, one of the largest and most comprehensive dexterous grasping datasets to date, comprising 1.4K grasp trajectories (including both successful and failed attempts) on 100 objects performed by humans and multiple robotic hands. The dataset is enriched with synchronized multi-view and first-person video streams, high-precision 3D motion capture, and high-resolution tactile signals. Notably, HRDexDB aligns human and robotic multimodal data under identical objects and similar grasping actions, providing high-accuracy spatiotemporal 3D ground truth. This alignment establishes a foundational benchmark for cross-domain policy learning and human-to-robot dexterous manipulation transfer.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Multi-modal Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We present HRDexDB, a large-scale, multi-modal dataset of high-fidelity dexterous grasping sequences featuring both human and diverse robotic hands. Unlike existing datasets, HRDexDB provides a comprehensive collection of grasping trajectories across human hands and multiple robot hand embodiments, spanning 100 diverse objects. Leveraging state-of-the-art vision methods and a new dedicated multi-camera system, our HRDexDB offers high-precision spatiotemporal 3D ground-truth motion for both the agent and the manipulated object. To facilitate the study of physical interaction, HRDexDB includes high-resolution tactile signals, synchronized multi-view video, and egocentric video streams. The dataset comprises 1.4K grasping trials, encompassing both successes and failures, each enriched with visual, kinematic, and tactile modalities. By providing closely aligned captures of human dexterity and robotic execution on the same target objects under comparable grasping motions, HRDexDB serves as a foundational benchmark for multi-modal policy learning and cross-domain dexterous manipulation.

Problem

• dexterous grasping
• human-robot comparison
• multi-modal dataset
• tactile sensing
• 3D motion capture

Innovation

• dexterous grasping
• multi-modal dataset
• 3D motion capture
• tactile sensing
• cross-domain manipulation