TLDR: This research evaluates how large language models (LLMs) can better process lengthy electronic health records (EHRs) for clinical tasks. It compares Retrieval-Augmented Generation (RAG), which retrieves only relevant text, against providing LLMs with very long inputs. The study found that RAG often matches or exceeds the performance of using recent notes and approaches full-context performance while using significantly fewer tokens, proving its efficiency for tasks like extracting imaging procedures and generating antibiotic timelines. However, its effectiveness varied for more subjective tasks like diagnosis generation.
Electronic Health Records (EHRs) are a treasure trove of patient information, but their sheer volume and often redundant nature pose a significant challenge for clinicians. Imagine a patient’s chart being as long as “Moby Dick”—navigating such extensive documentation to find critical details for diagnosis and treatment can be overwhelming. This challenge is what researchers are trying to tackle with the help of Large Language Models (LLMs).
LLMs show great promise in extracting and reasoning over unstructured clinical text, potentially easing the burden on healthcare professionals. However, even the most advanced LLMs have limitations on how much text they can process at once, known as their “context window.” This means that a full patient chart often exceeds what an LLM can handle in a single go.
Retrieval-Augmented Generation: A Smart Solution
One innovative approach to overcome this limitation is Retrieval-Augmented Generation (RAG). Instead of feeding the entire EHR into an LLM, RAG systems intelligently retrieve only the most relevant passages for a specific task. This not only reduces the number of input tokens required, making the process more efficient and cost-effective, but also helps mitigate the “lost-in-the-middle” effect, where important information gets overlooked within very long texts.
A recent research paper, titled “Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs,” explores the effectiveness of RAG compared to simply providing LLMs with very long inputs. The study, conducted by researchers including Skatje Myers, Dmitriy Dligach, and Timothy A. Miller, aimed to create clinical tasks that are easily replicable across different health systems without extensive manual annotation. You can find the full research paper here: Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs.
Three Key Clinical Tasks
To thoroughly evaluate RAG, the researchers designed three distinct clinical tasks, each requiring different levels of reasoning:
- Imaging Procedures: This task involved extracting structured information about diagnostic imaging (like X-rays, CT scans, MRIs) including the modality, date, and anatomical location from clinical notes. This is a relatively straightforward extraction task.
- Antibiotic Timelines: More complex, this task required identifying therapeutic antibiotic use for patients with severe infections and generating timelines of their administration. It involved not just identifying antibiotics but also understanding the medical context for their use.
- Diagnosis Generation: This was the most challenging task, requiring the model to identify key diagnoses relevant to a hospitalization, focusing on those that required active management and impacted the care plan, rather than just listing all mentioned diagnoses.
Key Findings: RAG’s Performance
The study tested three state-of-the-art LLMs (o4-mini, GPT-4o-mini, and DeepSeek-R1) using varying amounts of clinical context, including RAG-selected passages and direct inputs of recent notes or full context windows up to 128K tokens.
For the Imaging Procedures task, RAG demonstrated significant performance improvements across all models and evaluation methods. It achieved performance very close to using the LLMs’ full context window, but with a drastically smaller number of input tokens. This suggests that for tasks requiring precise information extraction, RAG is highly efficient and effective.
In the Antibiotic Timelines task, RAG consistently outperformed a rule-based baseline and showed performance comparable to using large amounts of recent notes. The gains from increasing the amount of retrieved text were slight, indicating that a limited number of targeted passages were sufficient to reconstruct the necessary temporal history. One interesting limitation noted was that sometimes the complete information needed for the “gold standard” timeline wasn’t present in the EHR due to patient transfers from other hospitals.
The Diagnosis Generation task proved to be the most challenging. Unlike the other two tasks, RAG did not consistently show improvement over simply using comparable amounts of recent notes. The overall performance for this task remained relatively flat and low across all models and data selection approaches. This suggests that the subjective nature of diagnosis generation and potential limitations in the evaluation method might have capped performance.
Also Read:
- Unlocking Better Clinical Predictions with Advanced AI Training
- Navigating AI’s Voice in Healthcare: A Deep Dive into Decoding Strategies for Medical Text Generation
Conclusion and Future Outlook
The research concludes that Retrieval-Augmented Generation remains a highly competitive and efficient approach for clinical reasoning over EHRs, even as newer LLMs become capable of handling increasingly longer texts. RAG consistently matched or approached the performance of full-context inputs while requiring significantly fewer tokens, especially for tasks like imaging procedure extraction and antibiotic timeline generation.
While the study highlights RAG’s value, it also points to areas for future work, such as further tuning retrieval parameters (queries, embedding models) and exploring additional clinically relevant tasks. The researchers also acknowledge limitations, including the inability to release the datasets due to privacy concerns and the inherent challenges in evaluating subjective tasks like diagnosis generation.
