Why Report Failed Interactions With Robots?! Towards Vignette-based Interaction Quality

2025-08-14

AI Summary

Existing HRI research inadequately documents system failures—particularly rare, context-dependent ones—undermining transparency and depth in interaction quality assessment. To address this, we introduce the ethnographic vignette method to human-robot interaction for the first time, systematically constructing a narrative, multidisciplinary, context-sensitive failure analysis framework grounded in the authors’ firsthand experiences with real-world HRI system failures. We propose a reusable vignette authoring methodology that successfully identifies and articulates previously overlooked interaction flaws and emergent behaviors. This approach overcomes key limitations of conventional quantitative evaluation—namely, its lack of explanatory power and contextual sensitivity. Our work enhances the visibility and reproducibility of failure cases, thereby facilitating more rigorous discourse on robotic capability boundaries and fostering cross-disciplinary understanding of HRI failure dynamics.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Although the quality of human-robot interactions has improved with the advent of LLMs, there are still various factors that cause systems to be sub-optimal when compared to human-human interactions. The nature and criticality of failures are often dependent on the context of the interaction and so cannot be generalized across the wide range of scenarios and experiments which have been implemented in HRI research. In this work we propose the use of a technique overlooked in the field of HRI, ethnographic vignettes, to clearly highlight these failures, particularly those that are rarely documented. We describe the methodology behind the process of writing vignettes and create our own based on our personal experiences with failures in HRI systems. We emphasize the strength of vignettes as the ability to communicate failures from a multi-disciplinary perspective, promote transparency about the capabilities of robots, and document unexpected behaviours which would otherwise be omitted from research reports. We encourage the use of vignettes to augment existing interaction evaluation methods.

Problem

• Highlighting failures in human-robot interactions using vignettes
• Documenting rare and unexpected HRI system failures
• Improving transparency and evaluation of robot capabilities

Innovation

• Ethnographic vignettes highlight HRI failures
• Vignettes document rare unexpected robot behaviors
• Multi-disciplinary failure communication via vignettes