Teaching Language Models To Gather Information Proactively

2025-07-28

AI Summary

Current large language models (LLMs) passively respond to incomplete or ambiguous prompts, lacking the capability to proactively identify information gaps and solicit critical missing information. Method: This paper introduces the “proactive information gathering” paradigm, proposing a scalable reinforcement fine-tuning framework: it synthesizes training data with partially masked task information to generate clarification questions, and designs a reward mechanism that incentivizes models to elicit users’ implicit knowledge. Using Qwen-2.5-7B as the base model, we train it to produce high-quality clarification queries. Contribution/Results: Automatic evaluation shows a +18% improvement over o3-mini; human evaluation yields 42% query quality and 28% preference rate for final output structure—both significantly surpassing baselines. The approach substantially enhances the depth of human–AI collaboration and solution quality.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large language models (LLMs) are increasingly expected to function as collaborative partners, engaging in back-and-forth dialogue to solve complex, ambiguous problems. However, current LLMs often falter in real-world settings, defaulting to passive responses or narrow clarifications when faced with incomplete or under-specified prompts, falling short of proactively gathering the missing information that is crucial for high-quality solutions. In this work, we introduce a new task paradigm: proactive information gathering, where LLMs must identify gaps in the provided context and strategically elicit implicit user knowledge through targeted questions. To systematically study and train this capability, we design a scalable framework that generates partially specified, real-world tasks, masking key information and simulating authentic ambiguity. Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information -- such as hidden domain expertise or fine-grained requirements -- that would otherwise remain unspoken. Experiments demonstrate that our trained Qwen-2.5-7B model significantly outperforms o3-mini by 18% on automatic evaluation metrics. More importantly, human evaluation reveals that clarification questions and final outlines generated by our model are favored by human annotators by 42% and 28% respectively. Together, these results highlight the value of proactive clarification in elevating LLMs from passive text generators to genuinely collaborative thought partners.

Problem

• LLMs fail to proactively gather missing information
• Need for strategic questioning to uncover implicit knowledge
• Enhancing LLMs from passive to collaborative partners

Innovation

• Proactive information gathering via targeted questions
• Reinforcement finetuning for eliciting implicit knowledge
• Scalable framework simulating real-world ambiguity