TLDR: TPS-Bench is a novel benchmark designed to evaluate how effectively AI agents plan and schedule the use of diverse tools to complete complex, multi-step ‘compounding tasks’. The research reveals that while LLMs are generally good at tool selection, their scheduling strategies vary, leading to trade-offs between task completion rates and operational efficiency (time and token usage). An initial study also demonstrates that reinforcement learning can significantly improve both the effectiveness and efficiency of AI agents in these challenging scenarios.
Large language models (LLMs) have shown impressive abilities in solving problems, from complex research to coding. However, a significant challenge remains: how well can these AI agents handle real-world problems that require using many different tools in a coordinated way? These are called ‘compounding tasks’, and they demand not just selecting the right tools but also strategically scheduling their use for maximum efficiency.
A new benchmark called TPS-Bench has been introduced to evaluate precisely this capability: an AI agent’s Tool Planning and Scheduling abilities. This benchmark aims to measure how effectively LLM agents can break down complex problems, choose the necessary tools from a large repository, and then decide the best order to execute those tools, including performing multiple actions simultaneously when possible.
The TPS-Bench dataset includes 200 compounding tasks, categorized into two difficulty levels: ‘Easy’ and ‘Hard’. These tasks are built around a diverse set of hundreds of ‘model context protocol’ (MCP) tools. The subtasks involved cover a wide range of activities, such as checking the weather, navigating maps, managing calendars, and performing web searches. For instance, a task might involve finding current weather, checking flight delays, recommending clothing, and searching for local attractions – all requiring careful planning and execution.
The evaluation of AI agents on TPS-Bench focuses on two main aspects: the task completion rate and efficiency. Task completion is assessed using an LLM-as-a-judge approach, which breaks down the task into subtasks and verifies their successful completion. Efficiency is measured by factors like the number of tokens used, the total execution time, and the number of ‘tool call turns’ (how many times the agent needs to interact with tools).
Initial studies on popular LLMs, including both closed-source models like GPT-4o and GLM-4.5, and open-source models like Qwen3-32B, have revealed interesting insights. Most models demonstrate reasonable tool planning capabilities, meaning they can generally pick the right tools. However, their scheduling strategies vary significantly. For example, GLM-4.5 achieved a high task completion rate (64.72% on hard tasks) but did so by making many sequential tool calls, leading to long execution times and high token usage. In contrast, GPT-4o prioritized parallel tool calls, resulting in faster execution and fewer tokens, but with a lower completion rate (45.08%). This highlights a clear trade-off between effectiveness and efficiency in current AI agents.
Further analysis included a ‘cost-of-pass’ metric, which considers both the monetary cost of using a model and its task completion rate. This metric provides a more holistic view of a model’s value, showing that a cheaper model with a very low completion rate might still be more expensive in the long run if it fails frequently.
Ablation studies were also conducted to understand the impact of tool selection and scheduling strategies. It was found that while different tool selection methods didn’t drastically change the task completion rate, they significantly affected efficiency, with ‘self-selection’ and ‘no selection’ strategies consuming more tokens and time. Regarding scheduling, serial execution generally led to higher task completion rates by allowing for more careful dependency evaluation, but at the cost of increased time and token usage, reinforcing the efficiency-effectiveness trade-off.
Intriguingly, the research also explored the use of reinforcement learning (RL) to improve scheduling efficiency. By training a smaller model, Qwen3-1.7B, with just 100 RL samples, researchers observed a 6% improvement in task completion rate and a 14% reduction in execution time on hard tasks. This suggests that RL can be a viable path to enhance an agent’s ability to balance performance and efficiency in complex, multi-tool scenarios.
Also Read:
- Diagnosing AI’s Reasoning Abilities with TempoBench
- Unveiling AI’s Research Prowess: A New Benchmark for LLM Agents
TPS-Bench represents a crucial step towards developing more capable and efficient AI agents for real-world applications. By providing a standardized way to evaluate tool planning and scheduling, it helps identify current limitations and paves the way for future advancements in agentic AI. You can find more details about this research paper here.
