TLDR: Researchers introduced PricingLogic, a new benchmark to test if Large Language Models (LLMs) can accurately calculate complex tourism prices with multiple, overlapping rules. They found that while LLMs show promise, especially with code-assisted reasoning, they still struggle significantly with harder pricing scenarios, highlighting a reliability gap for revenue-critical applications.
Large Language Models (LLMs) have shown incredible capabilities across many fields, from generating code to solving mathematical problems and following human instructions. However, deploying these powerful AI systems in real-world applications, especially those requiring high reliability and domain-specific knowledge, remains a significant challenge. One such area is tourism pricing, where complex, overlapping rules and dynamic policies make manual calculations labor-intensive and prone to errors.
Introducing PricingLogic: A New Benchmark for Tourism Pricing
A new research paper, PricingLogic: Evaluating LLMs Reasoning on Complex Tourism Pricing Tasks, introduces the first benchmark specifically designed to test whether LLMs can reliably automate tourism-related pricing. Developed in collaboration with travel agencies, PricingLogic aims to assess if AI can handle the intricate calculations involved in booking requests, which often include multiple destinations, varied fare types, and dynamic pricing policies.
The benchmark consists of 300 natural-language questions derived from 42 real-world pricing policies. These questions are categorized into two main difficulty levels:
- Basic Customer-Type Pricing: Involving straightforward pricing based on customer categories.
- Bundled-Tour Calculations: More advanced scenarios with interacting discounts and multiple feasible pricing options.
PricingLogic is further divided into three difficulty levels: simple, medium, and challenging, with increasing demands on reasoning and computational capabilities. Simple questions might involve a single customer type, while challenging ones present complex scenarios with large groups, diverse demographics, region-specific pricing, multiple attractions, and overlapping discount conditions.
How LLMs Were Evaluated
The researchers evaluated a range of LLMs, including proprietary models like GPT-4o and Claude Sonnet 4, and open-weight models such as DeepSeek-V3/R1 and Qwen2.5. Two primary approaches were used:
- End-to-End Prompting (E2E): This is the most straightforward method, where LLMs process pricing information in a single pass, identifying project details, visitor counts, and special conditions, then calculating prices based on applicable policies.
- Code-Assisted Reasoning (CaR): This two-stage approach involves LLMs first generating dedicated Python calculator functions for each pricing policy. In the second stage, the models parse natural language orders, extract relevant information, and convert it into arguments for the generated code, which then performs the calculation. A variant, CaR-Oracle, was also used, where human-written code was provided to isolate errors related to parameter extraction from those related to code generation.
Key Findings: A Steep Performance Drop on Harder Tasks
The evaluations revealed a significant performance drop for LLMs on the harder tiers of the PricingLogic benchmark. While models performed reasonably well on simple questions using direct prompting (E2E), their accuracy declined sharply as complexity increased. For challenging questions, even the best LLMs struggled to exceed 50% accuracy.
The Code-assisted Reasoning (CaR) approach generally improved accuracy across most models and difficulty levels. This suggests that separating policy interpretation from parameter extraction helps LLMs handle complex pricing logic more effectively. However, even with CaR, absolute performance remained below 60% for challenging questions, indicating considerable room for improvement. The study found that LLMs often struggle with identifying when bundled discounts should override other pricing and calculating the optimal combination when multiple valid bundle options exist.
Interestingly, even when provided with human-verified code (CaR-Oracle), models still faced difficulties, particularly with challenging tasks. This indicates that deep task comprehension and the ability to supply correct arguments to complex functions remain major bottlenecks, rather than just the ability to generate correct code.
Also Read:
- New Benchmark Unveils Large Language Models’ Challenges in Chinese Multi-Hop Reasoning
- Unmasking AI’s Reasoning Gaps: A New Benchmark for Logic Puzzles
Implications for AI in Revenue-Critical Applications
The findings highlight a critical gap between the general capabilities of today’s LLMs and the specific demands of revenue-critical applications like tourism pricing. Despite their advancements, LLMs are not yet reliably equipped to handle scenarios with multiple, overlapping fare rules without further safeguards or domain adaptation. The research underscores the importance of rigorous evaluation before deploying AI systems in financial contexts, where errors can lead to significant financial losses and erode customer trust.
