TLDR: A research paper systematically evaluated seven AI approaches, including statistical methods, domain-specific BERT models, general text models, and large language models (LLMs), to uncover disease interconnections using real-world clinical data (MIMIC-IV ICD-10 codes). The study found that while LLMs can identify existing relationships, they showed limited potential for discovering *novel* disease interconnections compared to other methods. The research also confirmed that medical-domain specific models learn beyond pure textual similarity and identified potential new disease associations, contributing a valuable medical disease ontology.
Understanding the intricate web of connections between different diseases is a cornerstone of medical research, crucial for improving diagnosis, treatment, and patient outcomes. Traditionally, identifying these interconnections has been a painstaking process, relying on manual analysis of vast clinical data. This approach is not only labor-intensive and subjective but also prone to disagreements among experts. While machine learning (ML) has emerged as a promising tool, it faces its own set of challenges, including selecting the best methods, determining the most reliable data sources (like electronic health records or structured disease descriptions), and the fundamental absence of a ‘ground truth’ for many unexplored disease relationships.
Large language models (LLMs), despite their broad utility in various fields, often lack the specialized medical knowledge required for such nuanced tasks. To bridge these gaps, a recent research paper titled “REVEALING INTERCONNECTIONS BETWEEN DISEASES: FROM STATISTICAL METHODS TO LARGE LANGUAGE MODELS” by Alina Ermilova, Dmitrii Kornilov, Sofia Samoilova, Ekaterina Laptenkova, Anastasia Kolesnikova, Ekaterina Podplutova, Senotrusova Sofya, and Maksim G. Sharaev, undertook a systematic evaluation of various approaches to uncover disease relationships. You can read the full paper here.
A Comprehensive Evaluation Framework
The researchers developed a robust framework that integrated seven distinct categories of methods, utilizing two primary data sources: sequences of ICD-10 codes from MIMIC-IV Electronic Health Records (EHRs) and the complete set of ICD-10 codes, both with and without their textual descriptions. The methods evaluated included:
- Real-world Data Approaches: This category involved statistical co-occurrence analysis, such as Fisher’s exact test and Jaccard similarity, along with a Masked Language Modeling (MLM) approach, all applied to actual clinical data.
- Domain-Specific BERT Variants: Models like Med-BERT and BioClinicalBERT, which are specifically trained on medical texts and data, were used to capture semantic relationships between diagnoses.
- General-Purpose Text Models: A standard BERT model and Yandex Doc Search were employed to assess interconnections based purely on textual similarities of disease descriptions.
- Large Language Models (LLMs): Four prominent LLMs – Mistral, DeepSeek, Qwen, and YandexGPT – were tested for their ability to predict disease co-occurrence patterns using carefully engineered prompts.
Key Findings and Insights
The study’s graph-based comparison of the resulting disease interconnection matrices yielded several significant insights. One of the most striking findings was that the LLM-based approaches produced interconnections with the lowest diversity of ICD code connections to different diseases compared to other methods. This suggests a crucial implication: LLMs, in their current form, may have limited potential for discovering truly novel disease interconnections, often reflecting existing knowledge rather than uncovering new associations.
Interestingly, methods within the same category did not always yield consistently similar interconnection patterns, highlighting the complexity and variability inherent in disease relationship discovery. Visual comparisons showed that Masked Language Modeling (MLM) and Med-BERT often produced similar patterns, while BioClinicalBERT frequently showed contrasting results. Text-based methods, as expected, exhibited uniformly high similarity scores, likely due to shared terminology among diseases within the same medical chapter.
Further analysis using Spearman correlations revealed relatively low associations between LLM-generated matrices and other methods, with correlations generally remaining below 0.12. While statistically significant, these weak correlations underscore that LLMs capture fundamentally different relationship patterns compared to domain-specific or statistical approaches.
Delving Deeper: Med-BERT and Novel Discoveries
The researchers also investigated two key hypotheses. First, they explored whether Med-BERT relies purely on textual data. By comparing ICD code pairs connected by pretrained BERT (which relies heavily on text) versus Med-BERT, they found that Med-BERT identifies relationships that go beyond mere textual similarity. This indicates that Med-BERT, trained on real patient ICD sequences, learns meaningful medical interconnections that are not solely derived from disease descriptions.
Second, the study examined whether certain interconnections represented noisy data in MIMIC-IV or potentially novel, previously unknown disease associations. By analyzing ICD code pairs statistically associated via Fisher’s exact test but not linked by Med-BERT and LLMs, they found that many of these represented genuine clinical patterns driven by shared risk factors and pathophysiological links, such as common comorbidities for lung and prostate cancer. Conversely, when looking at pairs linked by Med-BERT and LLMs but not by Fisher’s exact test, many had plausible clinical explanations (e.g., anatomical contiguity, metastatic spread). However, some connections, like those between lung cancer (C34) and stomach cancer (C16) or type 1 diabetes (E10), lacked clear mechanistic explanations in current literature, potentially representing underexplored or novel disease interconnections ripe for further clinical investigation.
Also Read:
- H-DDx: A Smarter Approach to Assessing AI’s Diagnostic Skills in Healthcare
- Large Language Models Reshape Topic Modeling Approaches
A Foundational Resource for Future Research
In the absence of comprehensive ground truth databases for medical interconnections, the results of this study constitute a valuable medical disease ontology. This ontology can serve as a foundational resource for future clinical research and artificial intelligence applications in healthcare, offering a structured view of disease relationships derived from diverse analytical methods.
The study acknowledges certain limitations, such as using MIMIC-IV, an ICU-only dataset that may not reflect the general population, and the complexities of ICD-10 coding that were not explicitly modeled. Future research directions include a more detailed investigation into potential noise in the MIMIC dataset, examining how identified interconnections affect downstream tasks like cancer prediction and mortality risk assessment, and incorporating unsupervised quality assessment for LLMs in medical contexts.
