TLDR: This research introduces a novel Retrieval Augmented Generation (RAG) chatbot designed to improve risk and quality assurance in highly regulated industries. By combining Large Language Models (LLMs) with hybrid search and relevance boosting, the system efficiently processes complex regulatory queries, reducing reliance on specialized experts. Evaluated on real-world queries, the deployed system demonstrates significant performance improvements and offers insights into hyperparameter optimization for RAG systems.
In highly regulated sectors like auditing, finance, and legal services, ensuring compliance with Risk Management & Quality (R&Q) standards is paramount. Employees frequently face the challenge of navigating intricate regulatory frameworks, handling numerous daily queries that demand precise interpretation of policies. Traditionally, this reliance on specialized experts often creates operational bottlenecks and limits the ability to scale operations effectively.
A new research paper, titled “Advancing Risk and Quality Assurance: A RAG Chatbot for Improved Regulatory Compliance,” introduces an innovative solution to this challenge. Authored by Lars Hillebrand, Armin Berger, Daniel Uedelhoven, David Berghaus, Ulrich Warning, Tim Dilmaghani, Bernd Kliem, Thomas Schmid, Rüdiger Loitz, and Rafet Sifa, the paper details a novel Retrieval Augmented Generation (RAG) system. This system leverages Large Language Models (LLMs), combined with hybrid search and relevance boosting, to significantly enhance the processing of R&Q queries.
The core of this system is a specialized chatbot powered by advanced AI capabilities. It’s designed to interpret user queries, retrieve the most relevant information from a vast knowledge base, and then generate accurate, contextually appropriate responses. A key innovation is its hybrid search strategy, which intelligently combines both vector similarity search (understanding the meaning of queries) and full-text search (matching keywords). The results from these two search methods are then re-ranked to ensure the most pertinent information is prioritized, further enhanced by a relevance boosting mechanism that prioritizes trusted internal documents.
The development of this RAG chatbot also included the creation of a robust evaluation framework. This automated system, utilizing tools like DeepEval and the G-Eval scoring method, assesses the chatbot’s performance based on correctness, completeness, relevance, and adherence to R&Q standards. The framework’s reliability was validated by comparing its LLM-based scores with manual expert evaluations across 124 responses, achieving a strong correlation. This rigorous evaluation demonstrated substantial improvements over traditional RAG approaches.
The system’s methodology involves three main components: an ingestion pipeline, the RAG chatbot itself, and the automated evaluation framework. The ingestion pipeline processes documents, parsing them into a structured data model, chunking them for context, and generating embeddings for efficient indexing. The chatbot then uses these indexed documents to answer queries, as described above. The prompt design for the chatbot is also sophisticated, including dynamic language detection and clear instructions for citing sources and avoiding ‘hallucinations’ (making up information).
Experiments conducted on a dataset of 124 expert-curated R&Q question-answer pairs revealed important insights. The research identified an optimal configuration that achieved the highest correctness scores for both answers and context. It was found that hybrid search consistently outperformed individual search methods, and relevance boosting further improved the prioritization of internal documents. Among the different LLM backbones tested, GPT-4o demonstrated the best overall performance, though all models delivered reasonable answers.
Also Read:
- Enhancing Developer Support with Adaptive AI Retrieval for Language Models
- Enhancing Drug Side Effect Detection with AI: A Look at RAG and GraphRAG Architectures
This RAG chatbot has been successfully deployed within the R&Q department of PricewaterhouseCoopers GmbH, showcasing its practical applicability and effectiveness in a real-world, highly regulated environment. The researchers believe this system offers valuable insights for practitioners looking to implement LLM-based chatbots in production. Future work aims to evolve the chatbot into a dynamic multi-agent system capable of more complex query dissection, clarifying questions, and multi-hop reasoning to further enhance its conversational capabilities. You can read the full research paper for more details here.
