TLDR: InfoMosaic-Bench is a new benchmark designed to evaluate how well AI agents can find and combine information from multiple sources, including both general web search and specialized domain tools. The research highlights that relying solely on web search is insufficient for complex tasks, and current LLM agents struggle with effectively using and integrating domain-specific tools, leading to inconsistent performance across different fields. The benchmark aims to drive improvements in tool-augmented AI for high-stakes applications.
Large Language Models (LLMs) have become increasingly powerful, transforming how we access and process information. However, a significant limitation of current LLM agents is their heavy reliance on open-web search. While the web offers a vast amount of information, it is often noisy, unreliable, and lacks the precise, domain-specific knowledge required for many real-world tasks.
The emergence of the Model Context Protocol (MCP) has opened doors for agents to interact with thousands of specialized tools. This development seemingly addresses the limitations of web-only search by providing access to curated, domain-specific data sources, such as biomedical databases, financial feeds, and mapping services. Yet, a crucial question remains: can LLM agents effectively leverage these diverse tools, and more importantly, can they integrate them seamlessly with general-purpose web search to solve complex, multi-source tasks?
To answer this, researchers have introduced InfoMosaic-Bench, the first benchmark specifically designed to evaluate multi-source information seeking in tool-augmented agents. This benchmark covers six representative domains: medicine, finance, maps, video, web, and multi-domain integration. It challenges agents to combine general-purpose search with specialized domain tools, ensuring that tasks require genuine multi-source reasoning.
The creation of InfoMosaic-Bench involved a scalable pipeline called InfoMosaic-Flow. This innovative system uses an organizer-worker architecture, where a central ‘organizer’ coordinates multiple domain-specific ‘workers’. This design ensures that task conditions are grounded in verified tool outputs, enforces dependencies across different information sources, and filters out simple cases that could be solved by trivial lookups. This rigorous approach guarantees both the reliability and non-trivial nature of the benchmark’s tasks.
Experiments conducted with 14 state-of-the-art LLM agents on InfoMosaic-Bench revealed several key findings:
Also Read:
- VitaBench: A New Standard for Evaluating LLM Agents in Real-World Scenarios
- Assessing AI’s Research Capabilities: Introducing Rigorous Bench
Key Findings from InfoMosaic-Bench:
-
Web Information Alone Is Insufficient: Even advanced models like GPT-5 achieved only 38.2% accuracy and a 67.5% pass rate when limited to web search. This clearly demonstrates that open-web search cannot meet the precise information needs of domain-specific tasks.
-
Domain Tools Offer Selective Benefits: While domain-specific tools improved performance in some areas, such as map and video domains, they sometimes degraded performance in others, like medical and finance. This suggests that current agents are still struggling to effectively exploit these specialized tools within each field.
-
Challenges in Tool Usage and Selection: A significant portion of failures (22.4%) stemmed from incorrect tool usage or selection. This highlights that LLMs currently lack the competence to reliably handle even basic tool interactions, indicating a fundamental gap in their ability to plan, select, parameterize, and time their tool calls effectively.
The benchmark dataset comprises 621 real-world samples and incorporates 77 distinct Model Context Protocol (MCP) tools across 7 servers. The findings from InfoMosaic-Bench expose a critical gap: while today’s models excel at web search, they are not yet capable of reliably exploiting domain-specific tools or combining them effectively. Addressing this limitation is crucial for deploying trustworthy agents in high-stakes fields like medicine, finance, and scientific discovery.
The researchers hope that InfoMosaic-Bench will encourage a shift from web-only search to a more principled and auditable multi-tool information-seeking approach, accelerating progress in these vital domains. For more details, you can refer to the original research paper.
The project’s code and dataset are available on GitHub: https://github.com/DorothyDUUU/Info-Mosaic
