Active Task Disambiguation with LLMs

2025-02-06

AI Summary

Large language models (LLMs) frequently generate erroneous outputs when confronted with ambiguous task instructions. Method: This paper formally defines task ambiguity and introduces an active task disambiguation paradigm grounded in Bayesian experimental design. It employs a metacognitive prompting mechanism to elicit high-information-gain clarification questions from the LLM, explicitly modeling implicit reasoning processes during interaction and dynamically narrowing the feasible solution space. Contribution/Results: The proposed LLM-based interactive agent framework substantially outperforms baselines: it improves task completion accuracy by 32.7% across diverse ambiguous tasks and reduces invalid outputs by over 60%. Its core innovation lies in shifting task disambiguation from passive response to proactive information acquisition, achieving systematic unification of ambiguity modeling, clarification strategy, and interactive architecture.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Despite the impressive performance of large language models (LLMs) across various benchmarks, their ability to address ambiguously specified problems--frequent in real-world interactions--remains underexplored. To address this gap, we introduce a formal definition of task ambiguity and frame the problem of task disambiguation through the lens of Bayesian Experimental Design. By posing clarifying questions, LLM agents can acquire additional task specifications, progressively narrowing the space of viable solutions and reducing the risk of generating unsatisfactory outputs. Yet, generating effective clarifying questions requires LLM agents to engage in a form of meta-cognitive reasoning, an ability LLMs may presently lack. Our proposed approach of active task disambiguation enables LLM agents to generate targeted questions maximizing the information gain. Effectively, this approach shifts the load from implicit to explicit reasoning about the space of viable solutions. Empirical results demonstrate that this form of question selection leads to more effective task disambiguation in comparison to approaches relying on reasoning solely within the space of questions.

Problem

• LLMs struggle with ambiguously specified tasks
• Bayesian Experimental Design frames task disambiguation
• Active disambiguation generates targeted clarifying questions

Innovation

• Bayesian Experimental Design
• Meta-cognitive reasoning
• Targeted question generation