Breaking the 15% Barrier: A Real-World Data-Driven System for Proactive Social Robot Triggered by User Nonverbal Cues

2026-07-13

AI Summary

This work addresses a critical limitation in current retail service robots, which predominantly rely on voice-only interaction and consequently fail to recognize nonverbal cues—such as approaching, waving, or pointing—resulting in approximately 15.3% of potential interactions being missed. To overcome this, the authors propose an end-to-end, nonverbal-driven dialogue framework that eliminates the need for handcrafted rules. The system employs a real-time, multi-target nonverbal behavior recognition module to detect user gestures and uses these detections as prompts for a large language model, thereby enabling proactive dialogue responses triggered by visual cues. By integrating vision and language models for joint reasoning, the approach significantly enhances response quality in nonverbal-initiated turns, as demonstrated in offline evaluations, and has been implemented in a real-time deployable prototype.

Abstract

Service robots in retail stores increasingly rely on cascaded speech pipelines (STT-LLM-TTS), yet many customer-robot interactions are initiated or guided by nonverbal behaviors such as approaching, waving, pointing, or showing items. This paper studies such cues in a real-world store deployment with a teleoperated humanoid robot and shows that a non-negligible portion of robot turns are triggered by nonverbal behaviors rather than spoken input, revealing a limitation of audio-only dialogue systems. In a 6-day in-the-wild deployment, 15.3\% of robot utterances were initiated by users' nonverbal behaviors rather than spoken input. Based on an analysis of observed customer behaviors, we define a set of frequent, service-relevant nonverbal cues and develop a real-time multi-person, multi-label recognizer that runs online from video. We then propose a dialogue framework that conditions LLM-based utterance generation on recognized nonverbal cue tokens, and optionally leverages a vision-language model when items are shown, enabling proactive robot responses without hand-crafted rules. We evaluate the approach offline on nonverbal-triggered turns and demonstrate an online prototype that reacts to users' nonverbal cues in real time.

Problem

• nonverbal cues
• service robots
• proactive interaction
• real-world deployment
• dialogue systems

Innovation

• nonverbal cues
• proactive robot interaction
• real-world deployment
• vision-language model
• multi-label recognition