TLDR: A new benchmark, INTERACTCOMP, reveals that advanced AI search agents struggle significantly with ambiguous user queries because they fail to interact and seek clarification. Despite having strong reasoning capabilities when given full context, models exhibit systematic overconfidence, leading to low accuracy (best at 13.73%) in real-world interactive scenarios. This highlights a critical stagnation in interaction capabilities compared to rapid improvements in general search performance, pointing to a major blind spot in AI development.
In the rapidly evolving landscape of artificial intelligence, language agents have shown remarkable prowess in tasks like web search and information retrieval. However, a recent research paper titled INTERACTCOMP: EVALUATINGSEARCHAGENTS WITHAMBIGUOUSQUERIES, authored by Mingyi Deng, Lijun Huang, Yani Fan, Jiayi Zhang, and a team of distinguished researchers, highlights a critical, often overlooked, limitation: these advanced search agents largely assume user queries are complete and unambiguous. This assumption, the researchers argue, diverges significantly from real-world scenarios where users frequently begin with incomplete or vague questions that require clarification through interaction.
The core problem identified is that most current search agents lack the interactive mechanisms necessary to engage in a dialogue to resolve ambiguity during the search process. When faced with an unclear query, agents tend to confidently commit to an assumed interpretation, often leading to incorrect answers and wasted computational resources. Existing benchmarks for search agents, such as GAIA and BrowseComp, typically provide all necessary information upfront, thus failing to assess an agent’s ability to proactively seek clarification.
Introducing INTERACTCOMP: A New Benchmark for Interactive Search
To address this crucial gap, the researchers introduced INTERACTCOMP, a novel benchmark specifically designed to evaluate whether search agents can recognize query ambiguity and actively interact with users to resolve it. The benchmark follows a principle of “easy to verify, interact to disambiguate.” It comprises 210 expertly crafted questions across nine diverse domains, constructed using a unique target-distractor methodology. This approach creates genuine ambiguity that cannot be resolved through search alone, necessitating interaction.
In an INTERACTCOMP task, an agent receives an ambiguous question. A simulated user holds the hidden context containing distinctive attributes needed for disambiguation. Agents can perform three actions: ‘search’ to retrieve web information, ‘interact’ to propose clarifying questions (which must be yes/no), and ‘answer’ to provide a final response. The simulated user responds with “yes,” “no,” or “I don’t know” based solely on the provided context. This setup forces agents to recognize ambiguity, gather disambiguating details through interaction, and then identify the correct answer.
Striking Findings: Overconfidence and Stagnation
The evaluation of 17 different models, including proprietary systems like GPT-5 and open-weight models like DeepSeek-R1, revealed a striking failure pattern. While models achieved strong performance (up to 71.50% accuracy) when provided with complete disambiguating context, the best model managed only 13.73% accuracy in the full interaction setting. This significant five-fold performance gap suggests that models fail not due to a lack of reasoning or search capabilities, but rather a systematic overconfidence that prevents them from engaging in necessary interaction.
Further analysis confirmed this diagnosis. Simply increasing the number of interaction opportunities from 5 to 20 rounds yielded minimal improvement, as models barely increased their questioning behavior. However, when models were *forced* to interact before answering, performance saw dramatic gains. For instance, GPT-5 doubled its accuracy from 20% to 40% when compelled to ask eight questions, demonstrating that the capability to interact effectively is latent but underutilized.
A longitudinal study tracking model development over 15 months revealed an even more concerning trend. While performance on benchmarks like BrowseComp (which assesses complex search with complete queries) improved seven-fold, INTERACTCOMP performance remained stagnant, clustering around 6-14% accuracy across all models. This stark divergence exposes a critical blind spot in AI development: rapid progress in search and reasoning has not translated into progress in interaction-based problem-solving.
Also Read:
- Unpacking LLM Long-Context Abilities: Insights from the LooGLE v2 Benchmark
- The Mirror Loop: Why AI’s Self-Reflection Can Lead to Stagnation
The Path Forward
The INTERACTCOMP benchmark serves as a vital resource for both evaluating and training interaction capabilities in search agents. The grounded nature of search tasks provides clear reward signals, making it well-suited for reinforcement learning approaches to develop agents that are uncertainty-aware and actively interactive. The research highlights that without an explicit focus on interaction, AI models will continue to advance in retrieval and reasoning while remaining primitive in recognizing ambiguity and seeking clarification – a significant limitation for their practical deployment in real-world scenarios. For more details, you can refer to the full research paper here.
