When Robots Rate Their Own Interactions: Engagement Validity and the Strangeness Failure

2026-06-22

AI Summary

This study addresses a critical gap in traditional human–robot interaction (HRI) evaluation, which relies solely on human-reported questionnaires and neglects the robot’s own evaluative perspective. The authors propose a novel “reverse evaluation” framework that enables an embodied robot (Nao) equipped with a large language model (LLM) to self-assess its interactions using established HRI scales, including HRI-CUES, Godspeed, and RoSAS, and compare these self-ratings against ground-truth human evaluations. Results indicate that the LLM achieves strong performance and high test–retest reliability (ICC ≥ .82) on engagement-related dimensions—such as satisfaction (r ≤ .65) and perceived enjoyment (r ≤ .72)—yet consistently exhibits inverse misjudgments on the uncanniness dimension (r = −.44 to −.67), revealing its inability to accurately infer internal affective states. These findings underscore the necessity of integrating multimodal signals, such as physiological and gaze data, to enhance the fidelity of robotic self-assessment.

Abstract

Human-robot interaction (HRI) evaluation relies almost exclusively on human-completed questionnaires, leaving the robot's perspective unexamined. We propose an \textit{inverted evaluation}, in which LLM-powered robots complete the same standardized instruments from their own perspective, and test whether these ratings agree with human ground truth. In Study~1, five LLMs completed HRI-CUES, Godspeed, and RoSAS questionnaires for 25~interactions ($N = 1{,}522$ evaluations) from the HRI-CUES dataset. LLMs achieved moderate-to-strong agreement on engagement dimensions (satisfaction $r$ up to $.65$ and enjoyment $r$ up to $.72$) with excellent test-retest reliability (ICC $\geq .82$), but \textit{systematically inverted} the comfort/strangeness dimension ($r = -.44$ to $-.67$, all $p < .05$), conflating engagement with comfort. In Study~2, a Nao robot running Claude~Sonnet~4.5 replicated these patterns in live interactions ($N = 4$), including real-time turn-by-turn assessment. The strangeness failure persisted across five models, synthetic controls, and embodied deployment for two participants. We argue that current LLM-based robots lack access to the internal affective states needed to assess constructs like strangeness, and that inverted evaluation requires supplementary modalities (e.g., physiological signals, gaze, proxemics) to move beyond behavioral proxies. These findings establish boundary conditions for using LLMs as interaction evaluators in HRI.

Problem

• human-robot interaction
• engagement validity
• strangeness
• LLM-based evaluation
• inverted evaluation

Innovation

• inverted evaluation
• LLM-based robots
• engagement validity
• strangeness failure
• human-robot interaction