Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody

2025-06-01

AI Summary

To address the ambiguity in spoken command interpretation caused by missing prosodic cues and the limited robustness of conventional ASR-dependent approaches, this paper proposes the first end-to-end prosody-driven spoken command intent parsing framework. It bypasses text transcription entirely, directly modeling prosodic features from raw speech to infer and resolve ambiguities. We introduce the first benchmark dataset of ambiguous spoken commands specifically designed for robotic task execution. Our method innovatively integrates a prosody-aware module with large language model-based in-context learning (ICL) to jointly perform multimodal alignment and task planning. Experiments demonstrate state-of-the-art performance: 95.79% accuracy in referential intent detection and 71.96% accuracy in selecting correct task plans for ambiguous commands—substantially enhancing robotic systems’ robustness and generalization in understanding natural, prosodically rich spoken instructions.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Enabling robots to accurately interpret and execute spoken language instructions is essential for effective human-robot collaboration. Traditional methods rely on speech recognition to transcribe speech into text, often discarding crucial prosodic cues needed for disambiguating intent. We propose a novel approach that directly leverages speech prosody to infer and resolve instruction intent. Predicted intents are integrated into large language models via in-context learning to disambiguate and select appropriate task plans. Additionally, we present the first ambiguous speech dataset for robotics, designed to advance research in speech disambiguation. Our method achieves 95.79% accuracy in detecting referent intents within an utterance and determines the intended task plan of ambiguous instructions with 71.96% accuracy, demonstrating its potential to significantly improve human-robot communication.

Problem

• Improving robot understanding of spoken instructions via prosody
• Resolving ambiguous speech intent using prosodic cues
• Enhancing human-robot communication with speech disambiguation

Innovation

• Leverages speech prosody for intent disambiguation
• Integrates intents into large language models
• Introduces ambiguous speech dataset for robotics