TLDR: A research paper details Fervento’s integration of Large Language Models (LLMs) into their CALEIDOHOTELS platform to improve the consistency and completeness of property descriptions. The study compared Mistral 7B (fine-tuned) and Mixtral 8x7B, finding Mixtral 8x7B superior in completeness, precision, and hallucination reduction, despite its significantly higher computational cost. The project, CaleidoGen, aims to enhance user experience, enable customization, and differentiate the platform through high-quality, consistent accommodation data.
The online travel industry, particularly property booking platforms, faces a significant challenge: maintaining consistent and up-to-date information about accommodations. Data often comes from various third-party providers, leading to incomplete or inconsistent details that can frustrate users and impact market share. To tackle this, Fervento, a software development startup, has integrated Large Language Models (LLMs) into its property reservation platform, CALEIDOHOTELS.
This initiative, detailed in the research paper Large Language Models in the Travel Domain: An Industrial Experience, explores the practical application of LLMs to enhance the quality and consistency of facility descriptions. The core idea is to leverage the LLMs’ ability to process and generate human-like text to fill data gaps and standardize property information across the platform.
The CaleidoGen System
Fervento developed a system called CaleidoGen, which uses Generative AI (GenAI) to create a coherent catalog of property descriptions from existing data sources. This system aims to reduce resource utilization compared to traditional methods and offers several key advantages:
-
Enhanced User Experience: By unifying descriptions across different providers, CaleidoGen ensures a consistent brand feel and simplifies comparisons for users.
-
Customization and Personalization: The system can highlight specific features or amenities, tailoring descriptions to seasonal interests or unique selling points.
-
Ease of Extension: A dedicated generation module makes it easier to integrate new or lower-quality data sources with minimal overhead.
-
Differentiation from Competitors: CALEIDOHOTELS can stand out with unique and easily updated property descriptions, turning its catalog into a strategic asset.
Evaluating LLM Performance
The researchers evaluated two prominent LLMs for this task: Mistral 7B and Mixtral 8x7B. Mistral 7B, a smaller and more resource-efficient model, was fine-tuned using the QLoRA technique. Mixtral 8x7B, a larger and more powerful model, was utilized with a refined system prompt. Both models were assessed on their ability to generate consistent and homogeneous descriptions while minimizing inaccuracies, often referred to as ‘hallucinations’.
The evaluation involved generating 20 facility descriptions for each model, using a distinct testing dataset to ensure unbiased results. The quality of the generated content was measured using several metrics:
-
Completeness: How much of the available context information was included in the description.
-
Precision: The factual accuracy of the information presented.
-
Hallucinations: The presence of fabricated or incorrect details.
-
Length of generation: The number of words in the generated text.
Also Read:
- Taming AI’s Tendency to Fabricate: A Deep Dive into Hallucination in Large Language Models
- Large Language Models Transform Recommender Systems: A Deep Dive into New Capabilities and Remaining Hurdles
Key Findings and Trade-offs
The results showed that both models performed well, but Mixtral 8x7B consistently outperformed Mistral 7B-FT (fine-tuned) across most metrics. Mixtral 8x7B achieved 99.6% completeness compared to Mistral 7B-FT’s 93%, and a precision of 98.8% versus 96%. Crucially, Mixtral 8x7B had a significantly lower hallucination rate of 1.2% compared to Mistral 7B-FT’s 4%, making its content more reliable. Although Mixtral 8x7B produced slightly shorter descriptions (averaging 249 words vs. 277), it did so without compromising essential details, indicating more concise output.
However, this superior performance came with a notable trade-off in computational cost. Mixtral 8x7B required approximately 50GB of VRAM and an hourly cost of about $1.61, roughly ten times more than Mistral 7B-FT, which needed only 5GB of VRAM and cost around $0.16 per hour. This highlights the balance between achieving high output quality and managing infrastructure and operational expenses.
Despite its higher computational demands, Mixtral 8x7B was ultimately chosen as the core model for CALEIDOHOTELS due to the critical importance of providing accurate and complete information in the travel domain. This industrial experience demonstrates the significant potential of advanced LLMs in solving complex data consistency challenges within the tourism and technology sectors.
The work also sets the stage for future developments, including LLM-driven features like generating personalized itineraries or guided tour plans based on user preferences and nearby attractions.
