REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery

2026-06-29

AI Summary

This work addresses the limitations of existing robot error-handling approaches, which typically treat failures as isolated events and rely on binary detection with rule-based recovery, thereby neglecting users’ dynamic perception and adaptation during real-world interactions. To bridge this gap, we introduce REPAIR-Bench, a benchmark derived from 214 interactions involving 41 users across four induced failure types, enriched with multimodal data including facial action units, head pose, speech, and emotional feedback. This benchmark enables cross-session failure detection, fine-grained classification, and prediction of user-preference-driven recovery strategies. We present the first evaluation framework encompassing the full failure lifecycle, incorporating longitudinal user adaptation modeling, non-binary failure categorization, and context-aware recovery mechanisms. Our proposed hierarchical recurrent network, combined with a QLoRA-finetuned Mistral-7B model, significantly outperforms baselines in cross-session failure detection (F1=0.80), temporal localization (median error: 2.97 seconds), and recovery strategy prediction (Hit@5=0.76).

Abstract

Understanding how users perceive and respond to robot failures is essential for building robust and trustworthy robot systems. Prior work, however, (i) often treats failures as independent events, (ii) emphasizes binary failure detection, (iii) with rule-based recovery modeling. We present REPAIR-Bench, built on 214 interaction trials from 41 participants, the benchmark spans four induced failure types and provides synchronized facial action units, head pose, speech transcripts, and post-interaction affect and recovery reports. The benchmark spans three novel evaluation tasks that jointly capture the lifecycle of failure in human-robot interaction (HRI): (i) failure detection over inter-dependent interaction sessions, modeling longitudinal user adaptation across repeated failures; (ii) visual failure-type classification beyond binary success/failure formulations; and (iii) user-centered recovery prediction, inferring users' preferred recovery strategies from interaction context rather than relying on manually designed or rule-based strategies. In baseline experiments, hierarchical recurrent modeling improved failure detection over a single-session model (strict F1: 0.80 vs. 0.68), achieved a failure localization mean signed error of -0.51 s, median absolute error of 2.97 s and, for recovery prediction, a QLoRA-tuned Mistral-7B reached Hit@5=0.76 and F1@5=0.32. REPAIR-Bench provides both the HRI and Medical HRI communities with a standardized framework for (1) evaluating robot failures and (2) building transparent, adaptive, and trustworthy recovery systems.

Problem

• robot failure perception
• human-robot interaction
• failure recovery
• user adaptation
• interaction benchmark

Innovation

• failure perception
• interaction recovery
• human-robot interaction
• multimodal benchmark
• user-centered recovery