RFM-HRI : A Multimodal Dataset of Medical Robot Failure, User Reaction and Recovery Preferences for Item Retrieval Tasks

2026-03-05

AI Summary

This study addresses the lack of systematic research on how interaction failures in healthcare robots affect user trust and task performance. Through a Wizard-of-Oz experiment conducted in both laboratory and hospital settings, the authors simulated four types of object retrieval failures and collected multimodal behavioral data—including facial action units, head pose, speech, and self-reported questionnaires—from 41 participants. They present the first publicly available multimodal dataset of interaction failures specifically designed for healthcare contexts. The findings reveal that such failures significantly reduce users’ emotional valence and perceived sense of control, eliciting confusion, irritation, and frustration, with emotional responses dynamically evolving across repeated failures. This work provides an empirical foundation and analytical framework for developing failure detection and recovery mechanisms in safety-critical human-robot interaction scenarios.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Interaction Techniques and Devices
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

While robots deployed in real-world environments inevitably experience interaction failures, understanding how users respond through verbal and non-verbal behaviors remains under-explored in human-robot interaction (HRI). This gap is particularly significant in healthcare-inspired settings, where interaction failures can directly affect task performance and user trust. We present the Robot Failures in Medical HRI (RFM-HRI) Dataset, a multimodal dataset capturing dyadic interactions between humans and robots embodied in crash carts, where communication failures are systematically induced during item retrieval tasks. Through Wizard-of-Oz studies with 41 participants across laboratory and hospital settings, we recorded responses to four failure types (speech, timing, comprehension, and search) derived from three years of crash-cart robot interaction data. The dataset contains 214 interaction samples including facial action units, head pose, speech transcripts, and post-interaction self-reports. Our analysis shows that failures significantly degrade affective valence and reduce perceived control compared to successful interactions. Failures are strongly associated with confusion, annoyance, and frustration, while successful interactions are characterized by surprise, relief, and confidence in task completion. Emotional responses also evolve across repeated failures, with confusion decreasing and frustration increasing over time. This work contributes (1) a publicly available multimodal dataset (RFM-HRI), (2) analysis of user responses to different failure types and preferred recovery strategies, and (3) a crash-cart retrieval scenario enabling systematic comparison of recovery strategies with implications for safety-critical failure recovery. Our findings provide a foundation for failure detection and recovery methods in embodied HRI.

Problem

• human-robot interaction
• robot failure
• user response
• healthcare robotics
• multimodal dataset

Innovation

• multimodal dataset
• human-robot interaction (HRI)
• failure recovery
• medical robotics
• affective response