TLDR: Judy is an AI chatbot developed to provide personalized outdoor trail recommendations using a Large Language Model (LLM) combined with Retrieval Augmented Generation (RAG). It addresses challenges of accuracy and usability in existing systems by integrating a trail database with conversational AI, demonstrating improved recommendation matching and efficiency through real-world case studies in Connecticut. The system leverages web-scraped data, advanced embedding models, and similarity search to deliver accurate and contextually rich responses for outdoor enthusiasts.
As more people seek the tranquility and adventure of outdoor recreational activities like hiking and biking, the demand for smart, personalized guidance on trails has grown significantly. Traditional methods, such as static online platforms or basic rule-based chatbots, often fall short in providing the detailed, conversational, and accurate information users need to plan their trips effectively.
Addressing these challenges, researchers have developed ‘Judy,’ an innovative outdoor trail recommendation chatbot. Judy leverages the power of Large Language Models (LLMs) combined with Retrieval Augmented Generation (RAG) to offer a more accurate, efficient, and user-friendly experience. The project focused on outdoor trails in Connecticut, USA, to gather concrete insights into its performance.
The development of Judy involved two main phases. The first phase, ‘Data Preparation & Preprocessing,’ focused on collecting and organizing essential outdoor trail information, including names, lengths, difficulties, and permitted activities. This data was then stored in a MySQL database hosted on Amazon RDS, ensuring streamlined access for the chatbot. The researchers used web scraping tools like Selenium and BeautifulSoup to gather trail features and reviews from platforms such as CT Trail Finder, Google Reviews, and TrailLink. These reviews underwent a thorough cleaning process to ensure consistency and relevance.
The second phase, ‘User Query & Recommendation with RAG,’ is where Judy truly shines. When a user poses a query about an outdoor trail, Judy’s LLM first interprets the request. If the query is straightforward and can be answered with structured data (like trail length or location), Judy generates an SQL query to fetch the information directly from the database. However, for more nuanced questions that require insights into user experiences or opinions – such as “what do people say about the scenery on Aldridge trail?” or “how crowded is the Pine Hill trail usually?” – Judy activates its RAG function.
The RAG function retrieves the most relevant trail reviews and their corresponding embeddings. The system evaluated different sentence embedding models, including Ollama (nomic-embed-text) and two types of pre-trained Sentence Transformers. The Sentence Transformer trained on question-answer pairs (multi-qa-mpnet-base-cos-v1) demonstrated the fastest response times, making it a key component for efficiency.
To determine the relevance of reviews, Judy employs Facebook AI Similarity Search (FAISS), which ranks reviews based on their similarity to the user’s query. The top relevant reviews, along with the original user question, are then fed into the LLM. This allows Judy to synthesize the information and generate comprehensive, contextually appropriate responses. For its core conversational abilities and natural language understanding, Judy utilizes Llama3, integrated with MySQL and FAISS through LangChain.
Experimental studies highlighted the effectiveness of Judy’s RAG-based approach. Judy achieved a recommendation matching accuracy of 96%, significantly outperforming an LLM-only version without RAG, which scored 88%. This improvement is attributed to RAG’s ability to retrieve specific, relevant reviews, preventing the LLM from being overwhelmed by processing large amounts of data. The research also explored the impact of ‘k’ – the number of top relevant reviews sent to the LLM – finding that a ‘k’ value of 5 offered an optimal balance between response time and accuracy.
To further enhance efficiency, especially given the observed longer response times with RAG, the team implemented caching for embeddings and reviews of queried trails. This practical enhancement helps speed up subsequent queries for the same trails.
Also Read:
- Enhancing Conversational Recommender Systems with Smart Data Augmentation
- Understanding Advanced AI Search Agents
The development of Judy provides valuable lessons in building LLM-based recommendation systems for real-world applications. Future work aims to integrate more diverse data sources, such as weather information and social network data, and conduct extensive user studies to gather feedback on Judy’s receptivity and acceptability. For more technical details on the project, you can refer to the full research paper here.
