TLDR: ProQ-BERT is a new transformer-based AI model that uses multi-modal electronic health records (EHR) to predict Chronic Kidney Disease (CKD) progression. It integrates demographic, clinical, and laboratory data, using a unique quantization method for continuous lab values. Evaluated on over 91,000 patients, ProQ-BERT significantly outperforms existing models like CEHR-BERT, achieving high accuracy (ROC-AUC up to 0.995) in predicting progression from CKD stage 3a to stage 5, especially for short-term outcomes. The model highlights the effectiveness of transformer architectures and temporal data handling in clinical prognosis.
Chronic Kidney Disease (CKD) is a widespread global health issue, affecting nearly 10% of the world’s population and often leading to severe health complications like end-stage renal failure. Predicting how CKD will progress is crucial for doctors to intervene early, manage resources effectively, and ultimately improve patient outcomes. Traditional methods for predicting CKD progression, often relying on statistical models or simpler machine learning techniques, frequently struggle to capture the intricate, time-sensitive relationships found in long-term patient health data.
Recent advancements in deep learning, particularly with transformer architectures, have brought a significant shift in how sequence-based data is modeled. These models, known for their self-attention mechanisms, are excellent at identifying long-range dependencies and contextual patterns. Unlike older recurrent neural networks, transformers can process information in parallel and dynamically weigh different features, making them highly suitable for the diverse and complex data found in electronic health records (EHRs), including laboratory results and demographic information.
A new study introduces a groundbreaking transformer-based framework called ProQ-BERT, designed to predict the prognosis of Chronic Kidney Disease. This innovative approach aims to significantly improve predictive accuracy and interpretability compared to existing methods. ProQ-BERT integrates various types of multi-modal electronic health records from the Seoul National University Hospital OMOP Common Data Model, combining demographic, clinical, and laboratory data.
The ProQ-BERT Approach: Integrating Diverse Data
One of the key innovations of ProQ-BERT lies in its unique way of handling continuous laboratory values. It employs a quantization-based tokenization method, where continuous lab results are converted into 10 discrete quantiles. This allows the model to effectively incorporate numerical laboratory data within its discrete token framework, building on ideas from previous foundational models like Ethos. The model also utilizes attention mechanisms, which contribute to its interpretability by highlighting which data points are most influential in a prediction.
The data used for this study comes from the Seoul National University Hospital Common Data Model (SNUH CDM), a comprehensive de-identified dataset spanning 20 years (2004-2024) and covering over 3.7 million patients. The researchers extracted information from six key domain tables: person, visit occurrence, condition occurrence, drug exposure, procedure occurrence, and measurement (which includes lab test results).
Defining the Patient Cohort and Outcome
The study cohort included 91,816 patients who met specific criteria, such as a CKD or ESRD diagnosis, an estimated glomerular filtration rate (eGFR) below 60mL/min/1.73 m² continuously for at least 90 days, or a urine albumin-to-creatinine ratio (uACR) above 30mg/g continuously for at least 90 days. Patients who never had an eGFR measurement at SNUH were excluded.
The primary outcome for prediction was defined as the progression of CKD from stage 3a to stage 5. An index date was assigned for each patient when their eGFR first fell below the threshold for stage 3a (<60mL/min/1.73m²) and persisted for 90 days. Worsening prognosis was then defined as reaching stage 5 (<15mL/min/1.73m²) within a specified follow-up period after the stage 3a index date.
How ProQ-BERT Processes Patient Data
The core of ProQ-BERT’s methodology involves transforming raw patient data into structured sequences of medical events. Each sequence begins with a special [CLS] token, followed by the patient’s gender. [SEP] tokens act as delimiters between different temporal segments, such as visits. Medical events like conditions, drugs, and procedures are converted into unique concept IDs and placed after their respective type tokens ([CONDITION], [DRUG], [PROCEDURE], [LAB]). To capture temporal relationships, time interval tokens (e.g., [TIME 5m-15m]) are inserted between consecutive events. Age tokens, calculated based on visit dates, are also included to track long-term disease progression.
Training and Evaluation
ProQ-BERT was first pretrained using a standard Masked Language Model (MLM) method. Following pretraining, it was fine-tuned for binary classification tasks, predicting whether a patient would progress to a specific CKD stage within various follow-up periods. For instance, one task involved predicting progression from stage 3a to stage 5 within 365 days.
The model’s performance was rigorously evaluated against CEHR-BERT, a well-established benchmark in electronic health record analysis. Evaluation metrics included Receiver Operating Characteristic-Area Under the Curve (ROC-AUC) and Precision-Recall-Area Under the Curve (PR-AUC), along with Accuracy, Precision, Recall, and F1 score. PR-AUC was particularly important due to the inherent class imbalance in CKD progression prediction.
Experiments were conducted across 15 different combinations of follow-up periods (180, 365, 730, 1095, and 1460 days) and assessment periods (180, 365, and 730 days) to test the model’s robustness.
Also Read:
- Benchmarking AI in Healthcare: Traditional Count-Based Models Hold Their Ground Against Advanced LLM Pipelines
- Enhancing Cancer Staging with Language Models: New Approaches for Interpretable Rule Induction
Remarkable Results
ProQ-BERT consistently outperformed CEHR-BERT across all evaluation metrics and temporal settings. For short-term prediction (180 days), ProQ-BERT achieved an impressive ROC-AUC of 0.995 and PR-AUC of 0.989, significantly surpassing CEHR-BERT’s 0.873 and 0.822 respectively. Even for longer-term predictions (1460 days), ProQ-BERT maintained strong performance with an ROC-AUC of 0.954 and PR-AUC of 0.915, compared to CEHR-BERT’s 0.837 and 0.813. The performance gap was most pronounced for shorter follow-up periods, indicating ProQ-BERT’s effectiveness in identifying early markers of rapid disease progression.
The model also demonstrated consistent high performance across varying historical data lengths (assessment periods), suggesting that it can make meaningful predictions even with different amounts of available historical data. The study concluded that while performance naturally decreased with longer follow-up horizons due to increased uncertainty, extending the assessment period generally improved predictive accuracy, highlighting the value of a broader temporal context.
In conclusion, ProQ-BERT represents a significant advancement in CKD prognosis prediction. Its success underscores the power of transformer-based architectures and the effective integration of diverse laboratory data representations. This research offers a promising direction for developing more personalized and effective care strategies for patients with Chronic Kidney Disease. You can read the full research paper here.
