TLDR: A study validated Sherpa Rx, an AI assistant using Retrieval-Augmented Generation (RAG) with pharmacogenomics data (CPIC guidelines, PharmGKB). It demonstrated high accuracy, relevance, completeness, and recall in answering clinical queries, outperforming other large language models like ChatGPT-4omini in specialized pharmacogenomic tasks and a real-world quiz. The research highlights the potential of AI in personalized medicine while acknowledging limitations and emphasizing its role as a decision-support tool.
The field of pharmacogenomics, which studies how an individual’s genetic makeup influences their response to drugs, is rapidly advancing. This area holds immense promise for personalized medicine, aiming to minimize adverse drug reactions and improve treatment outcomes by tailoring drug selection and dosing based on genetic profiles. The successful integration of pharmacogenomic data into clinical practice relies heavily on robust resources and innovative technologies.
Electronic health records (EHRs) and clinical decision support (CDS) systems are foundational in applying pharmacogenomic data. Resources like the Clinical Pharmacogenetics Implementation Consortium (CPIC) guidelines provide actionable recommendations for optimizing drug therapy based on genetic variants, while the Pharmacogenomics Knowledgebase (PharmGKB) compiles crucial pharmacogenomic data, including drug-gene associations and clinical guidelines.
To overcome barriers in the adoption of pharmacogenomics, artificial intelligence (AI) and machine learning (ML) are being increasingly utilized. Advanced models, particularly those supported by Retrieval-Augmented Generation (RAG), can enhance the accessibility and application of pharmacogenomic knowledge by embedding authoritative resources directly into their functionality. One such platform, Helix AI, offers specialized AI assistants like Sherpa Rx, designed to provide accurate pharmacogenomics insights by integrating resources such as CPIC guidelines.
Despite the potential, large language models (LLMs) have not been extensively tested in pharmacogenomics, leaving their ability to accurately interpret and apply guidelines largely unknown. To address this gap, a study was conducted to validate Sherpa Rx by evaluating its accuracy, completeness, and other key performance metrics when responding to pharmacogenomic queries. The ultimate goal is to enhance clinical decision-making, foster knowledge sharing, and promote equitable access to pharmacogenomic insights.
Methodology: Building and Testing Sherpa Rx
For this study, Sherpa Rx was specifically tailored for pharmacogenomic scenarios, focusing on queries related to CPIC guidelines. This approach combined retrieval-based and generative techniques to deliver contextually relevant and accurate responses, surpassing the capabilities of standalone generative models. The evaluation primarily targeted healthcare providers, recognizing their central role in utilizing pharmacogenomic insights, though a subset of queries also addressed adult and pediatric patients.
The core of Sherpa Rx’s knowledge base incorporated CPIC guidelines and PharmGKB data. In an initial phase (Phase 1), only CPIC guidelines were integrated. In a subsequent phase (Phase 2), PharmGKB data was added alongside CPIC guidelines to assess the impact of expanded resources on performance. This content was processed using the “text-embedding-3-small” model to generate embeddings, which were then stored and managed in a Pinecone vector database.
Prompt engineering was crucial in optimizing Sherpa Rx’s outputs for clinical relevance. The system used prompts defining the LLM’s role as a “Pharmacogenomics Specialist” to ensure clinically oriented responses. Guardrails were implemented through carefully designed prompts to maintain answer relevance and safety. The system performed a cosine similarity search to find the most relevant documents, with a temperature parameter set to 0 to ensure deterministic outputs.
A comprehensive dataset of 260 pharmacogenomic queries was developed, with 10 queries representing each of the 26 CPIC guidelines available at the time of the study. These queries were collaboratively created by a principal investigator and ChatGPT-4omini, ensuring diversity, clarity, and clinical meaningfulness.
The evaluation of Sherpa Rx’s performance and real-world applicability was conducted by a multidisciplinary team. Responses were assessed on a 5-point Likert scale for accuracy, relevance, completeness, and clarity. Recall was used for quantitative performance analysis, while precision and F1-score were calculated for a subset of queries.
Key Findings: Sherpa Rx’s Performance
The study demonstrated that the integration of a RAG approach with CPIC and PharmGKB data significantly improved the AI assistant’s accuracy and overall performance in handling pharmacogenomic queries. In Phase 1 (N=260 queries), Sherpa Rx achieved high average scores: accuracy of 4.9, relevance of 5.0, clarity of 5.0, completeness of 4.8, and recall of 0.99.
When comparing a subset of queries (N=20) across Phase 1, Phase 2 (with PharmGKB data), and ChatGPT-4omini, Phase 2 showed improved accuracy (4.6 vs. 4.4) and completeness (5.0 vs. 4.8) over Phase 1. Notably, ChatGPT-4omini achieved comparable scores for relevance (5.0) and clarity (4.9) but significantly lower scores in accuracy (3.9) and completeness (4.2). Phase 2 also achieved the highest average recall (0.99) compared to Phase 1 (0.97) and ChatGPT-4omini (0.85), indicating its superior ability to generate all relevant responses.
Statistical analysis confirmed that while accuracy generally increased in Phase 2, the difference from Phase 1 was not statistically significant. However, Phase 2 demonstrated significantly higher accuracy compared to ChatGPT-4omini.
In a 20-question quiz designed to assess real-world applicability, Sherpa Rx achieved an impressive 90% accuracy, outperforming ChatGPT-4omini (70%), Claude 3.7 Sonnet (85%), and Gemini 2.0 Flash (80%). This enhanced performance was attributed to the addition of a targeted query layer for drugs and genes of interest and an expanded context window, though this came at the cost of slightly slower performance.
Also Read:
- Beyond the Buzz: Understanding Large Language Models in Medicine
- Evaluating Retrieval-Augmented Generation in Diverse Knowledge Environments
Implications and Future Directions
The study underscores the importance of incorporating authoritative, domain-specific resources into AI models to enhance their ability to deliver actionable information in specialized fields like pharmacogenomics. The comparison between Phase 1 and Phase 2 suggests that integrating PharmGKB alongside CPIC guidelines provides value, highlighting opportunities for further refining data integration.
However, the study also acknowledged several limitations. Ensuring accurate interpretation is critical, as errors in pharmacogenomic data could compromise patient safety. Limitations included the presence of multiple correct answer choices in some quiz questions, small sample sizes for certain metrics, and the potential for bias due to a single reviewer. Future work should focus on enhancing data accuracy, consistency, and interoperability between CPIC, PharmGKB, and other sources like the Pharmacogene Variation Consortium (PharmVar). Implementing a hybrid search framework combining RAG with a fine-tuned pharmacokinetic model could further improve response accuracy.
It is crucial to emphasize that Sherpa Rx, or any similar AI tool, is designed to be a decision-support tool, complementing the expertise of healthcare providers rather than replacing clinical judgment. Healthcare professionals remain responsible for interpreting AI-generated recommendations within the context of individual patient needs. Prospective validation in clinical settings will be essential to demonstrate scalability, reliability, and provider adoption before widespread implementation.
This research highlights the transformative potential of generative AI in pharmacogenomics, demonstrating its capacity to improve decision-making and patient care by providing accurate, personalized responses. With further refinement and integration into healthcare systems, tools like Sherpa Rx can enhance provider productivity, support patient-centered care, and expand equitable access to pharmacogenomic data. You can read the full research paper here: Validating Pharmacogenomics Generative Artificial Intelligence Query Prompts Using Retrieval-Augmented Generation (RAG).
