When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search

2026-06-25

AI Summary

This work addresses the limited ability of existing search agents to proactively identify and effectively clarify ambiguities in vague, incomplete, or erroneous user queries, which often leads to deviated reasoning paths. To tackle this challenge, the authors introduce DiscoBench, a novel benchmark that systematically defines and annotates four realistic types of query ambiguity. They further develop a multi-turn interactive user simulator and a four-dimensional evaluation framework encompassing task utility, ambiguity recognition, interaction strategy, and cost efficiency. Experimental results reveal that current large language model–driven search agents exhibit significant deficiencies in ambiguity detection and clarification—so much so that blind retrieval can underperform naive guessing—thereby underscoring the critical need for and inherent challenges of proactive clarification in interactive search systems.

Abstract

Search agents powered by large language models (LLMs) are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.

Problem

• search agents
• ambiguity
• clarification
• deep search
• information-seeking tasks

Innovation

• clarification-aware search
• DiscoBench
• ambiguity detection
• interactive search agents
• deep search