TLDR: A research paper introduces Guide-RAG, a chatbot system designed to improve clinical question answering for complex, emerging diseases like Long COVID. By evaluating various Retrieval-Augmented Generation (RAG) corpus configurations, the study found that combining clinical guidelines with high-quality systematic reviews consistently outperformed both single-guideline approaches and large-scale literature databases. The system was evaluated using an LLM-as-a-judge framework across faithfulness, relevance, and comprehensiveness, utilizing a novel dataset of expert-generated Long COVID clinical questions (LongCOVID-CQ). The findings suggest that curated secondary reviews offer an optimal balance for AI chatbots in high-uncertainty medical domains, providing reliable and comprehensive guidance while avoiding information overload.
Artificial intelligence (AI) tools and large language models (LLMs) are increasingly being adopted in clinical medicine, with both patients and clinicians turning to chatbot platforms for medical information. While these systems offer timely guidance, they also carry risks such as hallucination, citation errors, and biases. This becomes particularly challenging for complex, heterogeneous, or poorly understood diseases like Long COVID (LC).
Long COVID affects an estimated 7% of U.S. adults, presenting as a multisystem condition with over 200 reported symptoms. It lacks a standardized diagnostic biomarker or evidence-based treatment, making clinical guidance often consensus-based and extrapolated from related syndromes. In such high-uncertainty settings, designing chatbots for clinical decision support is difficult, as relying on vast, unfiltered databases like PubMed or a single guideline can either overwhelm clinicians or miss evolving evidence.
To address these challenges, researchers developed and evaluated six Retrieval-Augmented Generation (RAG) corpus configurations for Long COVID clinical question answering. RAG systems aim to ground language model responses in external evidence, making the selection of corpus documents critical. The study proposes Guide-RAG, a chatbot system and evaluation framework that integrates both curated expert knowledge and comprehensive literature databases to effectively answer LC clinical questions.
Key Contributions of Guide-RAG
The Guide-RAG framework offers three main contributions:
-
Expert Corpus Curation: A targeted curation of a clinical guideline supplemented by three high-quality systematic reviews consistently outperformed both large-scale literature databases and narrow single-guideline approaches for Long COVID question answering using RAG.
-
Evaluation Metrics: The study adopted faithfulness, relevance, and comprehensiveness metrics specifically for Long COVID clinical applications. An LLM-as-a-judge framework was used to capture criteria directly relevant to clinical decision-making and trust-building.
-
LongCOVID-CQ: A specialized evaluation dataset of expert-generated Long COVID clinical questions was developed. These questions reflect the practical information needs that healthcare providers routinely encounter in patient care, targeting diagnosis, management strategies, and mechanisms.
Methodology and Evaluation
The researchers systematically evaluated six distinct corpus configurations within the Guide-RAG framework. These ranged from a control condition (standard GPT-4o without retrieval) to expert-curated corpuses and large-scale literature corpuses. The expert-curated options included a ‘Guideline only’ configuration (G-1), a ‘Guideline + systematic reviews’ configuration (GS-4), and a corpus comprising the 110 references cited within the guideline (R-110). Large-scale literature corpuses included a ‘PubMed corpus’ (PM) and ‘Web search’ (WS) using OpenAI’s GPT-4o capabilities.
The evaluation employed GPT-4o as an LLM-as-a-judge for pairwise comparisons across all configurations. Responses were assessed head-to-head based on faithfulness (content supported by retrieved documents), relevance (directly addressing the question), and comprehensiveness (thorough coverage of all aspects). An overall metric combined these three criteria equally.
Key Findings
The ‘Guideline + systematic reviews’ (GS-4) configuration consistently achieved superior overall performance, with win rates of 57.5-65% in pairwise comparisons. This configuration also ranked highest in faithfulness and comprehensiveness. Notably, despite using only four sources, GS-4 demonstrated greater comprehensiveness than both R-110 (110 references) and PM (PubMed corpus), which leveraged significantly larger corpuses.
While GS-4 excelled overall, the PubMed corpus (PM) showed a slight superiority in relevance, achieving win rates of 50-52.5% against other conditions. This suggests that PM’s expanded information access might enhance relevance performance.
A comparison between the ‘Guideline only’ (G-1) and ‘References of the guideline’ (R-110) configurations revealed that G-1 achieved a 57.5% win rate over R-110 in faithfulness and overall evaluation. This indicates that synthesized guidelines can provide enhanced faithfulness through better alignment with retrieved content, while still being comprehensive and relevant enough to maintain overall performance.
Expert clinical review further contextualized these quantitative results. It highlighted that corpora built from expert-curated sources consistently outperformed broader or unstructured collections. For instance, in symptom management, broader corpora sometimes produced superficially plausible but misleading or overconfident responses, often relying on single studies or speculative mechanisms. In contrast, the curated GS-4 corpus offered practical management guidance that acknowledged evidence gaps and provided more useful, responsible advice to clinicians.
Also Read:
- PokeeResearch-7B: Advancing AI Agents for Complex Research with Self-Correction
- Guideline Introduces Conversational AI Agent for Enhanced Advertising Intelligence
Implications and Future Directions
These findings suggest a crucial design principle for chatbots in emerging disease contexts: retrieval grounded in curated secondary reviews offers an optimal balance between narrow consensus documents and the unfiltered breadth of primary literature. This approach supports faithfulness, relevance, and comprehensiveness—qualities essential for decision support in high-uncertainty medical domains.
The LongCOVID-CQ dataset also addresses a critical gap in medical AI evaluation by providing clinically grounded questions that reflect real-world information needs, moving beyond traditional multiple-choice formats that test factual recall.
The study acknowledges limitations, including its focus on a single clinical domain (Long COVID), the use of a single LLM model (GPT-4o) as the evaluation judge, and a small, expert-generated question set. Future work will include human expert ratings, multiple model evaluations, and extending testing to other high-complexity medical domains. For more details, you can read the full paper here.
