TLDR: CostBench is a new benchmark designed to evaluate Large Language Model (LLM) agents’ ability to plan and adapt cost-optimally in dynamic environments. Focusing on travel planning, it uses tools with customizable costs and simulates real-world disruptions like tool failures or cost changes. The research found that even advanced LLMs like GPT-5 struggle significantly with cost-optimal planning, especially under dynamic conditions, revealing a major gap in their economic reasoning and adaptability.
Large Language Models (LLMs) have shown incredible abilities in reasoning, generating code, and solving complex problems. When these models are given external tools, they can interact with dynamic environments like the web or interactive systems, essentially becoming ‘agents’ that can perform multi-step tasks autonomously.
However, current evaluations of these LLM agents mostly focus on whether they can complete a task. They often overlook two critical aspects: how efficiently these agents use resources (their ‘economic reasoning’) and how well they can adapt their plans when the environment changes. This gap means we don’t fully understand if LLM agents can create and adjust plans to minimize costs in unpredictable real-world situations.
Introducing CostBench: A New Way to Evaluate LLM Agents
To address this, researchers have introduced CostBench, a new benchmark designed specifically to evaluate an agent’s economic reasoning and replanning skills. CostBench is a scalable, cost-focused framework set in the domain of travel planning. It presents agents with tasks that can be solved using various sequences of ‘atomic’ (single operation) and ‘composite’ (multiple operations chained together) tools, each with diverse and customizable costs.
What makes CostBench unique is its ability to simulate real-world unpredictability. It includes four types of ‘dynamic blocking events’ such as tool failures, changes in tool costs, shifts in user preferences, or the removal of certain tools. These events force agents to adapt and replan their strategies in real-time, making the evaluation much more realistic than static tests.
How CostBench Works
CostBench operates in two modes: a static mode, where conditions remain constant, and a dynamic mode, where user preferences, tool costs, and availability can change. In an iterative interaction loop, the agent observes the current task state, decides which tool to use, invokes it, and receives feedback, including the cost incurred. The goal is always to complete the task while minimizing the total accumulated cost.
The benchmark uses a structured four-stage workflow for travel tasks: preference identification, candidate search, candidate filtering, and final selection. Tools are designed with clear input-output schemas, ensuring that agents must follow logical steps. Each atomic tool has a randomized cost, and composite tools have costs based on their components plus some noise, preventing agents from simply memorizing optimal paths.
Key Findings: LLMs Struggle with Cost-Aware Planning
The evaluation of ten leading LLMs (both open-source and proprietary) on CostBench revealed significant weaknesses. Even the best-performing model, GPT-5, achieved less than a 75% exact match rate (meaning its tool-call sequence perfectly matched the optimal one) on the hardest static tasks. This performance dropped even further, to approximately 35%, under dynamic conditions involving cost changes.
Other models performed even worse, with some showing no significant improvement over a simple ‘greedy’ baseline that just picks the cheapest immediate option. This indicates a substantial gap in their ability to execute complete, cost-aware plans.
The study also found that:
- Performance consistently declined as task complexity increased.
- Models were highly sensitive to cost variations, performing better when cost differences between paths were more pronounced.
- Dynamic blocking events, especially ‘cost change’ and ‘ban tool’, caused the most significant degradation in performance. This suggests that implicit changes (like cost fluctuations) are particularly challenging because agents must first detect the change before replanning.
- Multiple blocking events severely undermined model robustness, with some models completely failing after just a few disruptions.
Also Read:
- Unpacking AI Agents’ Skills: A New Benchmark for Tool Planning and Scheduling in Complex Tasks
- Benchmarking LLMs for Cyber Threat Intelligence: Introducing AthenaBench
Looking Ahead
CostBench highlights that current LLM agents often lack sufficient ‘progress awareness’ – they struggle to track achieved subgoals or available inputs, leading to redundant or illogical tool uses. This limitation directly impacts their cost sensitivity and prevents truly cost-optimal planning.
The benchmark provides a crucial framework for future research, encouraging the development of LLM agents that are not only capable but also economically rational, resource-efficient, and resilient in complex, evolving real-world scenarios. To learn more, you can read the full research paper here.
