TLDR: MCPToolBench++ is a new large-scale benchmark for evaluating how well AI agents and Large Language Models (LLMs) use real-world tools via the Model Context Protocol (MCP). It addresses challenges like diverse tool types, varied response formats, and inconsistent tool success rates by providing a comprehensive dataset of single and multi-step tool calls across 40+ categories. The benchmark uses AST and Pass@K metrics to assess tool selection, parameter inference, and actual execution success, offering insights into model performance and common error causes in practical AI agent applications.
Large Language Models (LLMs) are becoming increasingly powerful, moving beyond just understanding and generating text to performing complex tasks by using various tools. These tools can range from search engines and web crawlers to financial data services and file systems. The integration of these diverse tools with LLMs is made possible through a standardized method called the Model Context Protocol (MCP).
However, evaluating how well LLMs and AI agents use these MCP tools presents several challenges. One major issue is the lack of comprehensive datasets or benchmarks that cover the wide variety of MCP tools available. Another difficulty arises from the diverse formats of responses generated by MCP tool calls, making consistent evaluation tricky. Unlike traditional tool-use benchmarks for programming or math, the success rate of real-world MCP tools isn’t guaranteed and can vary significantly across different MCP servers. Furthermore, the limited “context window” of LLMs restricts the number of tools they can effectively use in a single operation, as the textual descriptions and parameters of tools can consume a lot of token space.
To address these challenges, researchers have introduced MCPToolBench++, a new large-scale, multi-domain benchmark designed to evaluate AI agents’ ability to use MCP tools. As of July 2025, this benchmark incorporates over 4,000 MCP servers from more than 40 categories, gathered from MCP marketplaces and GitHub communities. The dataset includes both single-step and multi-step tool calls across various categories, providing a robust platform for evaluation.
The benchmark’s creation involves a sophisticated pipeline. It starts by collecting meta-information, configuration files, and tool schemas from open MCP marketplaces like smithery.ai and pulsemcp.com. A “Tool Sampler” then generates both single-step and multi-step tool call scenarios. For instance, a multi-step task might involve getting real-time stock prices, plotting a chart, and then calculating today’s change, requiring a sequence of tool calls. A “Query Generator” creates prompt templates and parameter values, even incorporating “Code Dictionaries” to handle specific inputs like stock ticker symbols or geocodes. Finally, a “Post-Processing & Validation” stage filters out low-quality or unreasonable queries, such as illogical travel routes or non-existent URLs.
The complexity of MCP function calls is significant, as LLMs need to process information from numerous servers, each offering multiple tools with detailed schema descriptions. To manage this, a “Tool Dispatcher” is crucial for retrieving only the most relevant tools, reducing the processing load on LLMs and improving accuracy.
MCPToolBench++ evaluates LLMs using two primary metrics: Abstract Syntax Tree (AST) accuracy and Pass@K accuracy. AST evaluates how well a model chooses the correct tool and infers its parameters based on the schema. Pass@K, specifically Pass@1, goes a step further by measuring whether the actual execution result from the MCP tool call aligns with the expected output. This includes checking if the tool call was successful, if input parameters were correct, and if the results were non-empty and accurate. The research highlights that a tool’s success rate significantly impacts the Pass@1 score, especially for tools requiring API connections.
The evaluation results show varying performance across different LLMs and categories. For example, Qwen3-coder performed well in Browser and Map categories for AST accuracy, while Qwen2.5-max excelled in File System and Finance. For Pass@1, Qwen3-coder led in Browser, Qwen2.5-max in File System, Claude-3.7-Sonnet in Search, GPT-4o in Map and Finance, and Kimi-K2-Instruct in Pay. Interestingly, the study notes that AST and Pass@K scores don’t always correlate positively. A model might have a high AST score (meaning it correctly identifies the tool and parameters) but a lower Pass@1 if the chosen tool frequently fails in real-world execution, or if it picks a less reliable tool when multiple options exist. This was observed in the Search category, where Claude-3.7-Sonnet’s higher Pass@1 was attributed to its more frequent selection of the reliable Google Custom Search Tool.
The paper also provides a detailed analysis of common error root causes for MCP tool call failures. These include “Parameter Errors” (incorrect or missing inputs), “API Errors” (issues with the tool’s underlying API), “Empty Result” (tool returns no data), and “Session & Runtime Errors.” Specific errors were identified for different domains, such as invalid longitude/latitude ranges for Map tools or “No Such File or Directory” for Browser use when saving screenshots. This detailed error analysis is crucial for future improvements in LLM agent development and MCP tool reliability.
Also Read:
- OmniBench-RAG: A New Standard for Evaluating Retrieval-Augmented Generation
- Enhancing AI Agent Tool Selection with Knowledge Graphs for Enterprise Tasks
This benchmark represents a significant step forward in evaluating the complex capabilities of AI agents in real-world tool-use scenarios. For more in-depth information, you can refer to the full research paper available here.
