Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking

2026-08-14

AI Summary

This study addresses the deficiency in missing information identification and proactive questioning within multi-turn information seeking by large language models. We formalize this task as a k-underdetermined constraint satisfaction problem and construct the MT-InfoSeek benchmark, introducing a Final Sufficiency metric to decouple information acquisition from answer generation evaluation. By quantifying interactive capabilities across three dimensions—what to ask, when to ask, and information utilization—we reveal that models systematically underestimate information gaps and exhibit sensitivity to query sequencing. Crucially, our findings demonstrate that information-seeking proficiency is distinct from question-answering ability. This work establishes a novel paradigm and evaluation benchmark for enhancing active interaction in large language models, providing critical insights into their current limitations in dynamic information gathering scenarios.

Abstract

When a user question is underspecified, a capable model should recognize that its context is insufficient, identify the missing information, ask for it, and respond only once that information determines a unique answer. We formalize multi-turn information seeking as solving a k-underspecified constraint satisfaction problem, where k is the number of variables jointly required to determine the target and therefore measures the degree of missing information. We instantiate the formulation in MT-InfoSeek, a controlled evaluation suite of 5,251 problems and 9,006 task instances spanning mathematics, logic, biology, medicine, and general knowledge. We evaluate models along three axes: what they ask, when they ask it, and how the acquired information affects the final answer. Performance degrades across models and domains as underspecification increases. Models recognize that additional information is needed but underestimate how much, and in logical problems at k = 2 they under-predict the degree of missing information about four times as often as they over-predict it. They also fail to identify a minimal sufficient set of queries, improve only marginally when given the true k, and often stop before acquiring sufficient information. In tasks with ordered dependencies, an incorrect query order reduces final accuracy even when the model eventually acquires all necessary information. We measure information seeking directly through final sufficiency, which records whether the acquired information determines the target independent of answer generation. This separation shows differences between models that final accuracy alone does not capture, and indicates that the ability to seek information over multiple turns is distinct from the ability to generate answers and is not measured by current LLM evaluations.

Problem

• Multi-Turn Information Seeking
• Underspecified Questions
• LLM Evaluation
• Constraint Satisfaction Problem
• Information Sufficiency

Innovation

• Multi-Turn Information Seeking
• Underspecified Constraint Satisfaction Problem
• MT-InfoSeek Benchmark
• Final Sufficiency Metric
• Information Acquisition Evaluation