TLDR: A new research introduces “proactive information gathering” for LLMs, where models identify knowledge gaps and ask targeted questions to elicit implicit user information. Using a synthetic conversation engine and reinforcement fine-tuning, their Qwen-2.5-7B model significantly outperforms baselines, demonstrating that training LLMs to proactively clarify leads to much better task completion and more collaborative interactions.
Large language models, or LLMs, are becoming increasingly vital for tackling complex tasks, from drafting proofs to debugging code. They excel when given clear, complete instructions. However, in real-world scenarios, users often provide incomplete or vague prompts, expecting the LLM to understand and fill in the gaps. Current LLMs frequently fall short here, offering passive responses or narrow clarifications instead of actively seeking the missing information crucial for high-quality solutions.
A new research paper introduces a novel approach called “proactive information gathering.” This paradigm challenges LLMs to identify what’s missing in a given context and strategically ask targeted questions to uncover implicit user knowledge. This is a significant shift from traditional clarification, which merely seeks to resolve ambiguities in existing input. Proactive questions, on the other hand, delve into unspoken dimensions, prompting users to provide new, complementary details that haven’t yet surfaced in the conversation. The goal is to transform LLMs from passive responders into genuine thought partners capable of eliciting relevant, actionable knowledge and guiding the dialogue towards more productive outcomes.
Designing LLMs that excel at this proactive information gathering presents several challenges. High-quality, collaborative dialogue data is scarce, and defining what constitutes a “good” proactive question can be subjective and difficult to quantify with standard reward signals.
To address these issues, the researchers developed a framework that specifically rewards pioneering, context-complementary questions—those that go beyond the provided information to proactively solicit critical details from users. Their approach incorporates three core innovations:
Task Formulation
They formalized proactive information gathering as a new task for LLMs, focusing on the ability to identify and elicit targeted, contextually missing information from users through dialogue.
Synthetic Conversation Engine
Leveraging an existing dataset called DOLOMITES, they built a simulation pipeline. This engine creates ambiguous prompts and rich clarification scenarios by systematically masking key information, making proactive questioning essential for task completion. It also simulates user responses based on this masked information, ensuring realistic interactions.
Also Read:
- Enhancing Conversational AI: A New Framework for Goal-Aligned User Simulators
- UserBench: A New Benchmark for Evaluating How AI Agents Understand User Needs
Reinforcement Fine-Tuning
A novel reward structure was proposed that rewards questions for reaching beyond the provided context. LLMs are then fine-tuned using a technique called Proximal Policy Optimization (PPO). This reward system focuses on whether a question elicits genuinely new, missing information, rather than just evaluating the final response.
Experiments demonstrated that their trained Qwen-2.5-7B model, referred to as Qwen-2.5-RFT, significantly outperformed other baseline models, including o3-mini, by 18% on automatic evaluation metrics. Importantly, human evaluations revealed that the clarification questions and final outlines generated by their model were preferred by human annotators by 42% and 28% respectively. This highlights the substantial value of proactive clarification in elevating LLMs from simple text generators to truly collaborative partners.
Further analysis showed that supervised fine-tuning (SFT) alone was insufficient to achieve robust proactive questioning capabilities. The performance gains from their method were particularly pronounced in open-ended domains like social science and humanities, which typically require deeper contextual reasoning and nuanced clarification. The Qwen-RFT model also demonstrated a clear ability to target both procedural and output-requirement segments of the context when asking questions, unlike other models that tended to focus on existing information.
This research marks a significant step forward in addressing a fundamental limitation of current large language models: their inability to proactively seek out missing, contextually relevant information in ambiguous, open-ended tasks. By transforming LLMs into active thought partners, this work paves the way for richer and more productive human-AI collaboration. You can read the full paper here.
