TLDR: This research compares large language models (Bert Base, BioBert, Bio+Clinical Bert, RoBerta, Clinical Longformer) for extracting contextualized medication event information from Electronic Health Records (EHRs) using the N2C2 2022 CMED dataset. It evaluates models on medication detection, medication event classification, and multi-dimensional context classification. Findings show that models pre-trained on clinical data excel at medication and event detection, while Bert Base (general domain) performs best for classifying the multi-dimensional context of medication events.
Electronic Health Records (EHRs) are a treasure trove of patient health information, containing everything from demographics and progress notes to medications and lab results. Extracting critical information from these unstructured notes manually is a monumental task, often prone to human error and inefficiency. This is where the power of Artificial Intelligence, specifically Natural Language Processing (NLP), comes into play, aiming to automate the extraction of vital clinical data.
A recent research paper, “Systematic Comparative Analysis of Large Pretrained Language Models on Contextualized Medication Event Extraction,” delves into the effectiveness of various large pretrained language models in understanding and extracting medication-related information from EHRs. Authored by Tariq Abdul-Quddoos, Xishuang Dong, and Lijun Qian, this study provides a comprehensive comparison of leading attention-based models on tasks crucial for clinical information extraction.
The Challenge: Understanding Medication Events
The research focuses on tasks from Track 1 of Harvard Medical School’s 2022 National Clinical NLP Challenges (n2c2), utilizing the Contextualized Medication Event Dataset (CMED). This dataset comprises unstructured EHRs and annotated notes designed to capture the nuanced context of medication changes in clinical narratives. The challenge aimed to develop robust solutions for three specific tasks:
- Medication Detection: Identifying mentions of medications within EHRs. This is a foundational step for any further medication-related analysis.
- Medication Event Classification: Determining if an identified medication mention is associated with an event (e.g., a change in dosage, start, or stop). Events are categorized as disposition, no disposition, or undetermined.
- Multi-dimensional Medication Event Context Classification: For medications with an associated event, classifying the context across five dimensions: action (e.g., Start, Stop, Increase), temporality (Past, Present, Future), certainty (Certain, Hypothetical), actor (Physician, Patient), and negation (negated, not negated).
The Models Under Scrutiny
The study fine-tuned and applied several prominent attention-based language models, each with different pre-training corpora and architectures:
- Bert Base: Pre-trained on general domain data like BooksCorpus and English Wikipedia.
- BioBert: An extension of Bert Base, further pre-trained on biomedical corpora such as PubMed Abstracts and PMC Full-text articles.
- Bio+Clinical Bert (two variations): Built upon BioBert, with additional pre-training on clinical notes from the MIMIC-III database. One variation used all MIMIC-III notes, while the other focused on discharge summaries.
- RoBerta Base: Shares Bert’s architecture but with modified pre-training, including longer training, bigger batches, removal of next sentence prediction, and dynamic masking.
- Clinical Longformer: Based on the Longformer architecture, which allows for processing much longer sequences than standard BERT models, with additional pre-training on clinical text from MIMIC-III.
These models were evaluated using standard metrics: precision, recall, and F1-Score, considering both strict and lenient matching for tasks 1 and 2.
Key Findings and Insights
The comparative analysis yielded interesting results, highlighting the importance of domain-specific pre-training:
- Medication Detection (Task 1): Models pre-trained on clinical data consistently outperformed those trained on general domain data. Specifically, Bio+Clinical Bert pre-trained on MIMIC-III Discharge notes achieved the highest performance, demonstrating a strict F-Score of 0.9355 and a lenient F-Score of 0.9669.
- Medication Event Classification (Task 2): Similar to medication detection, clinical data pre-trained models were superior. Clinical Longformer emerged as the top performer, with a strict F-Score of 0.8515 and a lenient F-Score of 0.8793.
- Multi-dimensional Medication Event Context Classification (Task 3): Surprisingly, for this more complex task of classifying the context of events, Bert Base, pre-trained on general domain data, showed the best performance. It achieved an overall F-Score of 0.7387 and a combined F-Score (where all context dimensions had to be correct) of 0.3006, significantly outperforming the domain-specific models. This suggests that while clinical pre-training helps with identifying entities and events, the more abstract contextual classification might benefit from broader linguistic understanding captured by general domain models, or perhaps the domain-specific models overfit to the specific clinical nuances for this task.
Also Read:
- Navigating the Future of Healthcare: A Deep Dive into Large Language Models in Medicine
- Optimizing Clinical Code Prediction: A Modular AI Framework for ICD-10-CM
Conclusion and Future Directions
This research underscores that while models pre-trained on clinical data are highly effective for detecting medications and medication events, a general domain model like Bert Base can be more effective for classifying the multi-dimensional context of these events. The study provides valuable insights for developing more effective NLP solutions in healthcare, particularly for extracting complex information from EHRs. Future work aims to improve performance on the multi-dimensional context classification task, especially by exploring data augmentation methods to address the scarcity of data for certain classes. You can read the full research paper here.
