TLDR: FinSearchComp is the first open-source benchmark for evaluating AI agents on realistic, expert-level financial search and reasoning tasks. Developed with 70 financial experts, it includes 635 questions across three task types: Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation. The benchmark reveals that while top models like Grok 4 are approaching human accuracy, significant gaps remain in handling time-sensitive data, multi-source reconciliation, and complex temporal reasoning. Equipping agents with web search and financial plugins substantially improves performance.
A new research paper introduces FinSearchComp, the first fully open-source benchmark designed to realistically evaluate how well AI agents can perform financial search and reasoning tasks. This benchmark aims to bridge the gap between current AI capabilities and the complex, real-world demands faced by financial analysts.
The paper highlights that while large language models (LLMs) are becoming core infrastructure for AI agents, evaluating their proficiency in specialized, time-sensitive domains like finance has been challenging. Existing datasets often fall short because they don’t capture the intricate, multi-step searches and knowledge-grounded reasoning that financial professionals routinely conduct. Constructing such realistic tasks requires deep financial expertise and up-to-date data, which is difficult to evaluate.
FinSearchComp addresses these challenges by offering 635 questions across global and Greater China markets. These questions are categorized into three main tasks that closely mimic actual financial analyst workflows:
Time-Sensitive Data Fetching
This task involves retrieving data that changes rapidly, such as current stock prices, exchange rates, or gold prices. It emphasizes quick retrieval and verification under strict time constraints, similar to how analysts monitor real-time market signals.
Simple Historical Lookup
These tasks require looking up fixed historical facts, like a company’s revenue in a specific quarter or the number of employees in a given year. The challenge here lies in aligning reporting conventions (e.g., fiscal year vs. calendar year) and ensuring data accuracy from historical disclosures.
Also Read:
- Evaluating AI Agent Architectures for Enterprise Workflows
- Improving RAG Performance with Hierarchical Document Chunking
Complex Historical Investigation
This is the most demanding task, involving multi-period aggregation and synthesis. Examples include identifying the month with the largest single-month increase in a major index over a long period or comparing revenue trends between different companies over several years. This requires stitching together multiple reports, checking consistency, and performing complex reasoning.
To ensure the benchmark’s difficulty and reliability, 70 professional financial experts were involved in annotating the data, and a rigorous multi-stage quality-assurance process was implemented. The evaluation of 21 models (products) on FinSearchComp revealed interesting insights. Grok 4 (web) performed best on the global subset, achieving near expert-level accuracy, while DouBao (web) led on the Greater China subset.
A key finding from the research is that equipping AI agents with web search capabilities and specialized financial plugins significantly improves their performance on FinSearchComp. The study also noted that the country of origin of models and tools can impact performance, with US models generally performing better on global assets and Chinese models excelling with Chinese assets, particularly for time-sensitive and simple historical lookups.
The paper concludes that FinSearchComp provides a professional, high-difficulty testbed for complex financial search and reasoning, offering an end-to-end evaluation that aligns with realistic analyst tasks. It highlights that while models like Grok 4 and GPT-5-Thinking are approaching human accuracy in certain areas, there are still significant gaps in handling freshness awareness, multi-source reconciliation, and temporal reasoning. This indicates that current AI systems remain fragile when faced with the full complexity of analyst-style tasks, underscoring the need for continued development in this critical domain. You can read the full paper here.
