TLDR: This research benchmarks three AI methodologies—count-based models, sequential transformers (CLMBR), and mixture-of-agents (MoA) LLM pipelines—for clinical prediction using structured electronic health records (EHR). The study found that count-based models remain highly competitive, often outperforming transformers and splitting wins with MoA pipelines across eight clinical tasks on the EHRSHOT dataset. MoA pipelines showed task-adaptive summarization capabilities, while recent patient history was most influential for count-based predictions.
A recent study delves into the effectiveness of various artificial intelligence models for predicting clinical outcomes using structured electronic health records (EHR). The research, titled “Count-Based Approaches Remain Strong: A Benchmark Against Transformer and LLM Pipelines on Structured EHR,” was conducted by Jifan Gao, Michael Rosenthal, Brian Wolpin, and Simona Cristea. This work provides a comprehensive comparison of traditional count-based models against more advanced transformer and large language model (LLM) pipelines, offering valuable insights into their performance on real-world clinical data.
Structured EHR data, which includes standardized medical concepts and relational tables, is crucial for clinical prediction. While count-based methods have historically performed well, there has been a growing interest in newer approaches, particularly those leveraging transformer architectures and LLMs. This study aimed to fill a knowledge gap by directly benchmarking these diverse methodologies.
Methodologies Explored
The researchers evaluated three main categories of models using the EHRSHOT dataset:
- Count-Based Models: These models aggregate the occurrence or frequency of medical concepts (like diagnoses and medications) into feature vectors. The study used LightGBM and TabPFN, a tabular foundation model, with ontology roll-ups and two time bins (recent and historical) to capture patient data.
- Pretrained Sequential Transformer (CLMBR): This approach treats EHR data as sequences of timestamped medical events. CLMBR learns contextual embeddings from these sequences, which are then used as input for lightweight classifiers like LightGBM and TabPFN.
- Mixture-of-Agents (MoA) LLM Pipeline: This innovative pipeline converts tabular patient histories into natural-language summaries using an LLM agent (Llama-3-8b or Qwen2.5-14B-Instruct). These summaries are then fed into a text classifier (BGE-large-en-v1.5 or ClinicalBERT) for prediction.
The EHRSHOT Dataset and Tasks
All analyses were performed on the publicly available EHRSHOT benchmark dataset, which includes predefined training, validation, and test splits. The study assessed eight different prediction tasks, covering both operational outcomes and new diagnostic predictions. Operational outcomes included Long Length of Stay, Readmission, and ICU Transfer. Diagnostic tasks involved predicting the first diagnosis of pancreatic cancer, hypertension, acute myocardial infarction (MI), hyperlipidemia, and lupus within a year after a hospital visit.
Key Findings
The head-to-head comparisons revealed that wins were largely split between the count-based and the mixture-of-agents methods across the eight evaluation tasks. Count-based models showed a slight overall edge and consistently outperformed the CLMBR transformer-based approach. For instance, in the earliest-label setting, count-based methods secured 5 out of 8 AUROC wins and 6 out of 8 AUPR wins, compared to MoA’s 3/8 AUROC and 2/8 AUPR wins. This suggests that despite their simplicity and interpretability, count-based models remain a strong contender for structured EHR prediction.
An interesting aspect of the MoA pipeline was its ability to produce task-adaptive intermediate summaries. The LLM agent would emphasize different medical concepts based on the prediction target. For example, for pancreatic cancer prediction, the summary amplified endocrine/metabolic and mental/behavioral content, while for ICU transfer, it elevated nervous system mentions. This adaptability demonstrates how LLMs can steer input information towards specific clinical prediction goals.
Further analysis using SHAP values for the LightGBM model showed that the most recent clinical history (within one year before prediction) carried the strongest signal for prediction, indicating that immediate events are often more influential than long-past ones.
Also Read:
- Intelligent AI System Enhances Medical Pre-Consultation with Proactive Inquiry
- Enhancing Cancer Staging with Language Models: New Approaches for Interpretable Rule Induction
Implications and Future Directions
The study concludes that count-based models remain a robust and practical choice for structured EHR prediction, even with the emergence of advanced LLM pipelines. While MoA pipelines offer gains on specific tasks and demonstrate impressive adaptability in generating clinically meaningful summaries, traditional methods still provide strong baselines. The researchers emphasize the need for future work to expand across multiple institutions, include a broader range of clinical information (like labs, vitals, and procedures), and test additional LLM agents and summarization strategies to further enhance these models.
For more detailed information, you can access the full research paper here.
