Robotic Task Ambiguity Resolution via Natural Language Interaction

2025-04-24

AI Summary

To address task execution failures in robotics caused by semantic ambiguity in natural language instructions, this paper introduces AmbResVLM—the first framework for active identification and resolution of task ambiguity. AmbResVLM integrates a vision-language multimodal large model (VLM), scene graph-based structured scene representation, uncertainty-aware language decoding, and a natural language interaction interface. It enables contextual grounding of task intent and explicit, interpretable reasoning about ambiguity, shifting language understanding from passive execution to an interactive, explainable intent clarification process. Evaluated on both simulation and real-world robotic platforms, AmbResVLM achieves significant improvements in ambiguity detection accuracy; downstream task success rates increase from 69.6% to 97.1%. All code, datasets, and models are publicly released.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness
• Computer Vision: Language and Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Language-conditioned policies have recently gained substantial adoption in robotics as they allow users to specify tasks using natural language, making them highly versatile. While much research has focused on improving the action prediction of language-conditioned policies, reasoning about task descriptions has been largely overlooked. Ambiguous task descriptions often lead to downstream policy failures due to misinterpretation by the robotic agent. To address this challenge, we introduce AmbResVLM, a novel method that grounds language goals in the observed scene and explicitly reasons about task ambiguity. We extensively evaluate its effectiveness in both simulated and real-world domains, demonstrating superior task ambiguity detection and resolution compared to recent state-of-the-art baselines. Finally, real robot experiments show that our model improves the performance of downstream robot policies, increasing the average success rate from 69.6% to 97.1%. We make the data, code, and trained models publicly available at https://ambres.cs.uni-freiburg.de.

Problem

• Resolving ambiguous task descriptions in robotics via natural language interaction
• Improving task ambiguity detection and resolution for language-conditioned policies
• Enhancing downstream robot policy performance through explicit ambiguity reasoning

Innovation

• Language-conditioned policies for versatile task specification
• AmbResVLM method for grounding and reasoning ambiguity
• Improves robot policy success rate significantly