TLDR: This research introduces a modular AI framework to improve the accuracy and efficiency of ICD-10-CM medical code prediction. It addresses challenges in large language model (LLM) application by proposing a novel LLM-as-judge evaluation for selecting the best base model, a redundancy-aware sampling method to optimize training data, and a section-aware fine-tuning approach that leverages structured clinical notes. Experiments across two Taiwanese hospitals demonstrate that this framework, particularly with the BioMistral model, significantly enhances coding performance and generalizability, offering a practical solution for automated medical coding systems.
Accurate medical coding, specifically using the International Classification of Diseases (ICD) system, is a cornerstone of clinical documentation, billing, and healthcare analytics. However, this task has traditionally been labor-intensive, prone to errors, and requires specialized expertise. While large language models (LLMs) have shown significant potential in automating this process, they face hurdles in selecting the most suitable base model, effectively using contextual information, and managing redundant training data.
A recent research paper, “Model selection meets clinical semantics: Optimizing ICD-10-CM prediction via LLM-as-Judge evaluation, redundancy-aware sampling, and section-aware fine-tuning”, introduces a new modular framework designed to tackle these challenges for ICD-10 Clinical Modification (ICD-10-CM) code prediction. Authored by Hong-Jie Dai, Zheng-Hao Li, An-Tai Lu, Bo-Tsz Shain, Ming-Ta Li, Tatheer Hussain Mir, Kuang-Te Wang, Min-I Su, Pei-Kang Liu, Ming-Ju Tsai, the study proposes a principled approach to enhance the accuracy and efficiency of automated medical coding.
A Smarter Way to Pick Models
One of the core innovations of this framework is its method for base model selection. Instead of relying on guesswork or extensive, costly fine-tuning of every candidate LLM, the researchers developed an “LLM-as-judge” evaluation protocol. This system uses a lightweight LLM to perform pairwise comparisons of candidate models, assessing their intrinsic understanding of ICD-10-CM code definitions. The results of these comparisons are then aggregated using the Plackett-Luce model to create a robust ranking, guiding the selection of the most appropriate base model for further fine-tuning. This approach significantly streamlines the initial model selection phase, making it more computationally efficient.
Refining Data for Better Training
Clinical notes often contain repetitive information, which can make training LLMs inefficient and sometimes even hinder their ability to generalize. To address this, the framework incorporates a redundancy-aware sampling strategy. This method uses embedding-based similarity measures to identify and remove semantically duplicated discharge summaries from the training data. By reducing redundancy, the approach improves training efficiency and helps the model generalize better without sacrificing the diversity of codes it learns. Experiments showed that this not only boosted performance but also cut training time by over 10%.
Understanding Context with Section-Aware Fine-Tuning
The research also leverages the structured format of electronic medical records (EMRs) from Taiwanese hospitals. These records are organized into distinct clinical sections like discharge diagnosis, medical history, and operation notes. The framework uses a “section-aware” prompting design during fine-tuning, allowing the LLM to understand how different combinations of these sections impact coding accuracy. The study found that incorporating more clinical sections, especially the Medical History, consistently improved prediction performance. A universal model was also developed, capable of handling diverse section combinations, demonstrating robust performance across varied input structures.
Real-World Validation
The effectiveness and generalizability of the proposed pipeline were rigorously tested across two distinct institutional datasets in Taiwan: Kaohsiung Medical University Chung-Ho Memorial Hospital (KMUH) and Taitung MacKay Memorial Hospital (TMMH). The results consistently showed that the selected base model, BioMistral, after fine-tuning, significantly outperformed baseline LLMs in both internal and external evaluations. The integration of redundancy-aware sampling and section-wise content enrichment further enhanced the model’s robustness and accuracy, proving its suitability for deployment in real-world hospital settings.
Also Read:
- Enhancing AI Trustworthiness: A New Approach to Curb Reward Hacking in Medical LLMs
- Scaling Qualitative Research: The HICode Approach to Deep Text Analysis
Looking Ahead
While the framework marks a significant step forward, the authors acknowledge limitations. The current work focuses solely on ICD-10-CM diagnostic codes, not procedure codes, and primarily uses discharge summaries. Future research could expand to other clinical document types and address the challenge of “coding drift” due to evolving clinical guidelines. Nevertheless, this study provides a practical, scalable, and institution-ready solution for automated medical coding systems, offering valuable insights for the future of AI-driven healthcare.
