From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping

2026-07-01

AI Summary

This study addresses the unresolved question of whether performance improvements in multimodal human–robot interaction systems are perceptible to users. Through a within-subjects user study, it directly links an increase in end-to-end task success rate—from 75% to 90%—to users’ perceived experience. The authors compare a baseline system integrating Whisper, Florence-2, LLaMA 3.1, and an interval type-2 fuzzy controller against an enhanced system employing Grounding DINO, SAM, and Qwen 3.5 9B in a tabletop object-grasping task. Results show that 70.83% of participants significantly preferred the enhanced system (p = 0.043), reporting substantial improvements in perceived speed, reliability, and fluency (all p < 0.001, with large to very large effect sizes), thereby demonstrating that technical advancements can effectively translate into tangible enhancements in user-perceived interaction quality.

Abstract

Improvements in the technical performance of human--robot interaction (HRI) systems do not automatically translate into differences that human users can detect during live interaction. This paper investigates whether a 15 percentage point gain in end-to-end task success (from 75% in a multimodal baseline system to 90% in an improved configuration identified through a prior ablation study) is sufficient to produce consistent and measurable differences in user perception. The baseline system combines Whisper for speech recognition, Florence-2 for open-vocabulary object detection, LLaMA 3.1 for action extraction, and an interval Type-2 fuzzy logic controller for motion execution. The improved configuration replaces the perception and language modules with Grounding DINO + SAM and Qwen 3.5 9B, respectively, while retaining the same controller. A within-subject user study with 24 participants compared both systems on the same tabletop object-grasping task. After interacting with each configuration, participants rated perceived speed, reliability, and overall competence and fluency on a 7-point Likert scale. Results show that 17 out of 24 participants (70.83%) preferred the improved system (exact binomial test, p = 0.043, h = 0.43), and all three perceptual constructs were rated significantly higher for the improved configuration after Holm correction, with large to very large effect sizes (p < 0.001). These findings confirm that the identified technical improvements are perceptible to users in direct interaction and underscore the importance of complementing benchmark evaluation with user-centred evidence when assessing robotic manipulation pipelines.

Problem

• human-robot interaction
• user perception
• object detection and grasping
• multimodal system
• task success rate

Innovation

• multimodal human-robot interaction
• user perception
• object detection and grasping
• perceptible system improvement
• user study