TLDR: A study compared the performance of five LLMs (GPT-3.5, Mistral-7b-instruct, LLaMa2-7b-chat, Falcon-7b-instruct, Orca-mini-v3-7b) in RAG-based question-answering within computer science literature. It found that RAG significantly reduces hallucination and provides up-to-date answers. GPT-3.5 with RAG performed best overall, while Mistral-7b-instruct was the top open-source LLM. The research highlighted a trade-off between cost (proprietary models) and latency (open-source models), emphasizing RAG’s crucial role in enhancing LLM reliability for domain-specific knowledge.
Large Language Models (LLMs) have transformed how we access information, moving beyond traditional search engines to provide direct answers. However, these powerful AI models often face challenges like generating incorrect information, known as ‘hallucination,’ and becoming outdated quickly since their knowledge is limited to their training data. For instance, many LLMs might not know about events or discoveries made after their last training update.
Enhancing LLMs with Retrieval Augmented Generation (RAG)
To overcome these limitations, a technique called Retrieval Augmented Generation (RAG) is gaining prominence. RAG allows LLMs to retrieve information from external, up-to-date databases specified by the user. This means an LLM can access the latest findings in a field, significantly reducing hallucination and ensuring the answers are current and accurate. This is particularly beneficial for rapidly evolving domains like computer science, where new research and theories emerge constantly.
A Deep Dive into LLM Performance with RAG
A recent study, titled Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature, conducted by Ranul Dayarathne, Uvini Ranaweera, and Upeksha Ganegoda, explored how different LLMs perform in question-answering tasks within computer science when supported by RAG. The research aimed to fill a gap in understanding RAG’s integration and performance specifically in this domain.
The study evaluated five popular LLMs: OpenAI’s GPT-3.5 and four open-source models – Mistral-7b-instruct, LLaMa2-7b-chat, Falcon-7b-instruct, and Orca-mini-v3-7b. The goal was to compare their effectiveness, efficiency, and cost when answering questions based on recent computer science literature.
How the Study Was Conducted
The researchers built a custom dataset using abstracts from computer science journal papers published between 2023 and 2024, focusing on topics like large language models, quantum computing, and edge computing. These abstracts were carefully selected to ensure they contained the most current discussions not typically included in the training data of older LLMs.
After cleaning and processing the abstracts, they were converted into numerical representations called ’embeddings’ using a specialized model called SPECTER, designed for scientific documents. These embeddings were then stored in a FAISS vector store, enabling efficient similarity searches for user queries. A Question-Answering (QA) pipeline was designed to take user queries, retrieve relevant information from the vector store, and then use an LLM to generate a concise answer. The pipeline also considered chat history to refine search queries for follow-up questions.
Performance was measured using several metrics: accuracy and precision for binary (Yes/No) questions, and cosine similarity, human expert ranking, and Google’s Gemini AI ranking for long-answer questions. The study also considered the latency (time taken to generate an answer) and the cost associated with each LLM.
Key Findings and Insights
The study revealed several important insights:
- GPT-3.5’s Strong Performance: When paired with RAG, GPT-3.5 consistently delivered the highest accuracy and precision for binary questions and the best cosine similarity scores for long-answer questions. This reaffirms its status as a highly advanced LLM, though it comes with a financial cost.
- Mistral-7b-instruct Leads Open-Source LLMs: Among the open-source models, Mistral-7b-instruct, combined with RAG, outperformed the others in both binary and long-answer questions. This is a promising result, indicating that open-source LLMs can be highly effective alternatives, especially considering they are free to use.
- Latency vs. Cost Trade-off: While open-source LLMs are free, they generally had higher average latencies (took longer to generate answers) compared to GPT-3.5. Orca-mini-v3-7b, however, reported the shortest average latency among the open-source models. This highlights a crucial trade-off: proprietary models offer speed and top-tier performance at a cost, while open-source models are free but might require better local infrastructure to match speed.
- RAG’s Indispensable Role: The study clearly demonstrated that LLMs without RAG (like a standalone ChatGPT instance) performed significantly worse on questions requiring recent knowledge. This underscores RAG’s critical role in enabling LLMs to provide accurate, up-to-date, and non-hallucinated answers, especially for domain-specific queries.
- Importance of Prompt Engineering: The research also noted that the way prompts are structured significantly impacts the quality of answers, especially for certain open-source models like Mistral-7b-instruct. Clear and precise instructions are essential for optimal outputs.
Also Read:
- AI Judges for E-commerce: Scaling Evaluation of Product Recommendations
- ChiMDQA: A New Comprehensive Dataset for Chinese Document Question Answering
The Future of RAG-Enhanced LLMs
This research highlights that integrating RAG with LLMs is a powerful approach to enhance their capabilities, particularly for accessing and synthesizing recent scholarly information in fields like computer science. While proprietary models like GPT-3.5 offer superior performance, open-source alternatives like Mistral-7b-instruct are proving to be highly capable and cost-effective, especially with optimized infrastructure. As LLMs continue to evolve, RAG will remain a vital technique for ensuring they provide reliable, current, and contextually relevant information.
