TLDR: A new framework uses open-source large language models (LLMs) like Qwen and MedGemma to automatically extract and standardize medication prescription attributes (e.g., drug name, quantity, duration) from diverse Electronic Health Record (EHR) systems. This enables consistent calculation of “MOUD days” (medication coverage duration) for opioid use disorder treatments, overcoming data heterogeneity. The system achieves high accuracy (over 90% for top models) and coverage, offering a scalable, privacy-preserving alternative to traditional data processing, with ongoing work to expand its scope and refine performance.
Healthcare systems face a significant challenge in managing and analyzing medication data, especially when it comes to monitoring treatments for opioid use disorder (MOUD). Electronic Health Record (EHR) systems, while vital, often store prescription details in various formats, making it difficult to get a clear, consistent picture of a patient’s medication history across different clinics. This inconsistency can hinder effective treatment monitoring and quality improvement efforts.
A new framework has been developed that leverages open-source large language models (LLMs) to tackle this problem head-on. The goal is to extract key medication attributes like prescription date, drug name, duration, total quantity, daily quantity, and refills from diverse EHR systems. By standardizing this information, the framework can then calculate a crucial metric: “MOUD days,” which represents the total medication coverage duration for a patient.
How the System Works
The process begins by taking raw, varied EHR data from different clinics. An instruction-tuned LLM acts as a central translator, converting this messy data into a clean, unified JSON format. This structured output is crucial because it ensures consistency and makes the data much easier to work with. The system uses a technique called constrained generation, which forces the LLM to produce output that strictly follows a predefined structure, minimizing errors and ensuring data quality.
After the LLM extracts the data, a post-processing stage refines it further. This involves normalizing data types (like converting dates and quantities), applying rules to catch logical inconsistencies (e.g., ensuring total quantity isn’t less than daily quantity), and filtering out nonsensical values. Finally, the “MOUD days” are calculated. This calculation prioritizes explicitly stated durations, but if that’s missing, it derives the duration from the total and daily quantities, providing a comprehensive measure of medication coverage.
Choosing the Right AI Models
Due to strict privacy regulations like HIPAA, the researchers focused on open-source LLMs that could be deployed locally, avoiding reliance on external, closed-source APIs. They evaluated several models, including different versions of Llama, Qwen, Gemma, and MedGemma, to find the best balance between performance and efficiency. Larger models generally performed better, with Qwen2.5-32B and MedGemma-27B showing particularly strong results.
Performance and Insights
The framework was tested on over 25,000 prescription records from 1,257 patients across five clinics participating in a national MOUD study. A benchmark of 10,369 manually annotated records served as the ground truth for evaluation. The performance was measured by “coverage” (the percentage of records successfully processed) and “record-level exact match accuracy” (how perfectly the extracted attributes matched the ground truth).
Results showed significant variation across clinics, highlighting the inherent heterogeneity of EHR data. However, for records that were successfully processed, the models achieved high accuracy. Qwen2.5-32B stood out with an impressive 93.4% overall coverage and 93.0% exact-match accuracy. MedGemma-27B also performed very well, achieving 93.1% coverage and 92.2% accuracy.
During the evaluation, common errors were identified and addressed. For instance, missing dosage information was imputed using typical values for the same drug. Special handling was implemented for monthly or weekly injectable medications (like Vivitrol), where the duration was set based on the documented schedule. Unit checks were also added to prevent mass units (e.g., “250 g”) from being mistakenly read as daily counts.
Also Read:
- CMOMgen: Automating Complex Ontology Alignment with Pattern-Guided AI
- MedAlign: A New AI Framework for Accurate and Efficient Medical Imaging Analysis
Impact and Future Directions
This innovative framework offers a robust and scalable alternative to traditional, often brittle, data extraction methods. By enabling consistent, cross-site analysis of MOUD exposure, adherence, and retention, it can significantly accelerate research and improve patient care, especially in underserved rural communities. The ability to deploy these models locally also ensures data privacy and security.
While promising, the researchers acknowledge limitations, such as the need to validate the framework’s generalizability across an even wider range of EHR systems and to extend its application beyond MOUD prescriptions. Future work will involve expanding evaluations, benchmarking against more advanced reasoning models, exploring efficient fine-tuning techniques, and developing user-friendly interfaces for expert review. You can read the full research paper for more details here: Research Paper on LLM-based Medication Extraction.
