A Model-Agnostic Approach for Semantically Driven Disambiguation in Human-Robot Interaction

2024-09-25

AI Summary

In large shared environments (e.g., households), semantic ambiguity in user instructions leads to object localization ambiguity for robots; existing approaches rely on single-shot visual detection or static positional prediction, failing to handle partial-understanding scenarios. This paper proposes a model-agnostic, semantics-driven iterative clarification framework that integrates knowledge embedding with multi-turn informative questioning, compatible with arbitrary semantic encoders and large language models. Through human–robot collaborative clarification, the framework progressively resolves ambiguities to achieve more accurate first-attempt object localization. Experiments demonstrate significant improvements in initial localization accuracy across diverse ambiguous instruction benchmarks. Furthermore, we publicly release ExpressionDataset—a high-quality, human-annotated expression dataset—establishing a new benchmark and resource for research on ambiguous instruction understanding.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Ambiguities are inevitable in human-robot interaction, especially when a robot follows user instructions in a large, shared space. For example, if a user asks the robot to find an object in a home environment with underspecified instructions, the object could be in multiple locations depending on missing factors. For instance, a bowl might be in the kitchen cabinet or on the dining room table, depending on whether it is clean or dirty, full or empty, and the presence of other objects around it. Previous works on object search have assumed that the queried object is immediately visible to the robot or have predicted object locations using one-shot inferences, which are likely to fail for ambiguous or partially understood instructions. This paper focuses on these gaps and presents a novel model-agnostic approach leveraging semantically driven clarifications to enhance the robot's ability to locate queried objects in fewer attempts. Specifically, we leverage different knowledge embedding models, and when ambiguities arise, we propose an informative clarification method, which follows an iterative prediction process. The user experiment evaluation of our method shows that our approach is applicable to different custom semantic encoders as well as LLMs, and informative clarifications improve performances, enabling the robot to locate objects on its first attempts. The user experiment data is publicly available at https://github.com/IrmakDogan/ExpressionDataset.

Problem

• Resolving ambiguities in human-robot interaction instructions
• Improving robot object search with semantic clarifications
• Enhancing location prediction for underspecified user queries

Innovation

• Model-agnostic approach for disambiguation
• Semantically driven iterative clarifications
• Leverages knowledge embeddings and LLMs