TLDR: TripTide is a new benchmark designed to test how well Large Language Models (LLMs) can adapt travel itineraries when unexpected disruptions like flight cancellations or attraction closures occur. It considers disruption severity and traveler preferences, using new metrics to evaluate how LLMs preserve original intent, respond to changes, and maintain semantic, spatial, and sequential coherence in revised plans. The research shows LLMs can adapt but still face challenges in complex, real-world scenarios, highlighting the need for more robust AI in travel planning.
Travel planning can be exciting, but real-world journeys rarely go off without a hitch. Unexpected events like flight delays, hotel issues, or attraction closures can quickly turn a dream trip into a nightmare. While Large Language Models (LLMs) have shown promise in generating personalized travel itineraries, they often struggle to adapt these plans when disruptions occur. This is where a new benchmark called TripTide steps in.
TripTide is the first benchmark specifically designed to evaluate how well LLMs can adapt travel itineraries when faced with realistic disruptions. It addresses a crucial gap in current LLM capabilities by simulating various real-world scenarios, considering factors like the severity of the disruption and the traveler’s tolerance for changes.
The benchmark introduces a comprehensive framework for understanding disruptions. These can range from transport issues like flight cancellations, accommodation problems such as unsafe locations, restaurant closures, or attraction-related disruptions like a museum being shut for maintenance. TripTide categorizes these disruptions by severity: step-level (affecting a single activity), day-level (impacting a whole day’s plan), and plan-level (requiring major itinerary overhauls).
Beyond disruptions, TripTide also models different traveler profiles and their tolerance for change. For instance, a “Flexi-Venturer” is open to rerouting and substitutions, while a “Plan-Bound” traveler prefers minimal changes and strict adherence to the original itinerary. This personalization allows for a more nuanced assessment of LLM responses, ensuring that revised plans not only address the disruption but also align with the user’s preferences.
Also Read:
- OutboundEval: Advancing AI Performance in Intelligent Outbound Calling
- Bridging the Gap: How Symbolic AI Enhances Transparency and Reasoning in Large Language Models
Evaluating Adaptive Travel Planning
To assess LLMs, TripTide proposes a suite of novel evaluation metrics. The “Preservation of Intent” metric checks if the revised plan still meets the traveler’s original goals and preferences. “Responsiveness” measures how promptly and appropriately the LLM addresses the disruption. Finally, “Adaptability” metrics quantify the semantic (thematic consistency), spatial (geographic convenience), and sequential (order of activities) changes between the original and modified itineraries.
Experiments conducted with models like GPT-4o, Qwen2.5-7B-Instruct, and Phi-4-mini Instruct revealed interesting insights. GPT-4o generally maintained strong semantic fidelity and modest spatial reorganization, especially for longer trips, though its ability to mitigate disruptions slightly decreased with increasing plan duration. Qwen2.5-7B-Instruct showed higher responsiveness in longer itineraries but struggled more with semantic and spatial coherence. Phi-4-mini Instruct often failed to deliver accurate plans but correctly identified disruptions.
Human evaluations by domain experts confirmed that LLMs, particularly GPT-4o, are good at detecting disruptions and making corrective edits. They often performed “smart swaps” by replacing a closed attraction with a similar, nearby alternative. The models also showed an understanding of “human factors” by sometimes adding rest time after strenuous activities and improving logistics like adding appropriate transportation for larger groups. However, struggles were noted in “superficial fixes” that didn’t truly resolve the issue, missing root causes of problems, overlooking real-world timing constraints, and failing to propagate “ripple effects” of local adjustments throughout the entire plan.
TripTide is built upon an augmented version of the TripCraft dataset, now including 1,000 travel planning queries across 3, 5, and 7-day durations, each with a disruption query and a human-annotated revised plan. This extensive dataset, with over 11,000 possible disruptions, provides a high-fidelity environment for evaluating LLMs’ adaptive planning capabilities.
This benchmark sets a new standard for evaluating and improving LLM-driven travel planning systems, emphasizing adaptability, personalization, and resilience in the face of real-world uncertainties. For more in-depth information, you can read the full research paper here.
