OpenRoboCare: A Multimodal Multi-Task Expert Demonstration Dataset for Robot Caregiving

2025-11-17

AI Summary

Current robotic caregiving research is hindered by the absence of large-scale, diverse, expert-annotated multimodal datasets, limiting progress in occlusion-aware perception, safe physical interaction, and long-horizon task planning. To address this, we introduce the first expert-driven, multimodal demonstration dataset specifically designed for caregiving scenarios. Five activities of daily living were performed by licensed occupational therapists, with synchronized acquisition of RGB-D video, full-body pose, eye-tracking trajectories, fine-grained action annotations, and tactile signals. This dataset systematically captures clinical-grade caregiving strategies—previously undocumented in robotics benchmarks—and enables multimodal fusion modeling and behavioral analysis. Empirical evaluation reveals that it poses substantial challenges to state-of-the-art perception and activity recognition models. The dataset thus establishes a scalable benchmark and analytical framework for advancing robot safety, adaptability, and reliable human–robot collaboration in assistive settings.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We present OpenRoboCare, a multimodal dataset for robot caregiving, capturing expert occupational therapist demonstrations of Activities of Daily Living (ADLs). Caregiving tasks involve complex physical human-robot interactions, requiring precise perception under occlusions, safe physical contact, and long-horizon planning. While recent advances in robot learning from demonstrations have shown promise, there is a lack of a large-scale, diverse, and expert-driven dataset that captures real-world caregiving routines. To address this gap, we collect data from 21 occupational therapists performing 15 ADL tasks on two manikins. The dataset spans five modalities: RGB-D video, pose tracking, eye-gaze tracking, task and action annotations, and tactile sensing, providing rich multimodal insights into caregiver movement, attention, force application, and task execution strategies. We further analyze expert caregiving principles and strategies, offering insights to improve robot efficiency and task feasibility. Additionally, our evaluations demonstrate that OpenRoboCare presents challenges for state-of-the-art robot perception and human activity recognition methods, both critical for developing safe and adaptive assistive robots, highlighting the value of our contribution. See our website for additional visualizations: https://emprise.cs.cornell.edu/robo-care/.

Problem

• Lack of large-scale expert dataset for robot caregiving demonstrations
• Need for multimodal data capturing complex human-robot care interactions
• Challenges in robot perception and activity recognition for assistive tasks

Innovation

• Multimodal dataset capturing expert therapist demonstrations
• Includes RGB-D video, pose, gaze, annotations, and tactile data
• Provides insights for robot perception and activity recognition