RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation

2024-12-18

arXiv.org

AI Summary

To address the scarcity of large-scale, high-quality, and diverse manipulation datasets for multimodal robot imitation learning, this work introduces the first benchmark dataset collected on a unified platform: it encompasses 96 object categories, 479 distinct tasks, 107k successful demonstrations, and 5k failure cases annotated with causal attributions, accompanied by an Isaac Sim digital twin environment. We propose a standardized teleoperation paradigm compatible with four distinct robotic platforms—Panda, UR5e, AgileX dual-arm, and humanoid dual-dexterous-hand—integrating multi-view vision, proprioceptive encoding, natural language task descriptions, and failure attribution labels. A digital twin–enabled closed-loop evaluation framework is established. The dataset enables robust Vision-Language-Action (VLA) model training, yielding significant improvements in cross-task generalization and manipulation success rates. To date, it represents the largest consistent benchmark for multimodal robot imitation learning.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

In this paper, we introduce RoboMIND (Multi-embodiment Intelligence Normative Data for Robot Manipulation), a dataset containing 107k demonstration trajectories across 479 diverse tasks involving 96 object classes. RoboMIND is collected through human teleoperation and encompasses comprehensive robotic-related information, including multi-view observations, proprioceptive robot state information, and linguistic task descriptions. To ensure data consistency and reliability for imitation learning, RoboMIND is built on a unified data collection platform and a standardized protocol, covering four distinct robotic embodiments: the Franka Emika Panda, the UR5e, the AgileX dual-arm robot, and a humanoid robot with dual dexterous hands. Our dataset also includes 5k real-world failure demonstrations, each accompanied by detailed causes, enabling failure reflection and correction during policy learning. Additionally, we created a digital twin environment in the Isaac Sim simulator, replicating the real-world tasks and assets, which facilitates the low-cost collection of additional training data and enables efficient evaluation. To demonstrate the quality and diversity of our dataset, we conducted extensive experiments using various imitation learning methods for single-task settings and state-of-the-art Vision-Language-Action (VLA) models for multi-task scenarios. By leveraging RoboMIND, the VLA models achieved high manipulation success rates and demonstrated strong generalization capabilities. To the best of our knowledge, RoboMIND is the largest multi-embodiment teleoperation dataset collected on a unified platform, providing large-scale and high-quality robotic training data. Our project is at https://x-humanoid-robomind.github.io/.

Problem

• RoboMIND provides multi-embodiment robot manipulation data.
• Includes 107k trajectories for 479 diverse tasks.
• Facilitates imitation and Vision-Language-Action model training.

Innovation

• Multi-embodiment teleoperation dataset
• Unified data collection platform
• Digital twin environment simulation