TLDR: A study evaluated various Large Language Models (LLMs) for classifying hierarchical ICD-10 codes from clinical discharge summaries. It found that while reasoning-based LLMs generally outperformed non-reasoning models (with Gemini 2.5 Pro showing the highest performance), no model achieved high accuracy (F1 scores below 57%), especially at more granular code levels. The research suggests LLMs can assist clinical coders but are not yet ready for autonomous deployment, highlighting the need for further development in domain-specific fine-tuning and hybrid models.
The healthcare industry relies heavily on accurate clinical coding, especially for classifying hierarchical ICD-10 codes from patient discharge summaries. This process is crucial for billing, healthcare planning, and research. Traditionally, human coders perform this labor-intensive and error-prone task. However, the emergence of Large Language Models (LLMs) offers a promising avenue for automation or augmentation of this critical function.
A recent study, “Evaluating Hierarchical Clinical Document Classification Using Reasoning-Based LLMs,” delves into the capabilities of various LLMs in this complex domain. The research aimed to benchmark a diverse set of LLMs, including both reasoning and non-reasoning models, to understand their effectiveness in classifying hierarchical ICD-10 codes and to evaluate the impact of structured reasoning on their performance. The full research paper can be found here: Research Paper Link.
Study Methodology
The researchers utilized the MIMIC-IV dataset, selecting 1,500 discharge summaries labeled with the top 10 most frequent ICD-10 codes. To prepare the data for LLM input, a clinical NLP tool called cTAKES was used to preprocess the summaries, extracting only clinically relevant tokens like medications, symptoms, and diseases. This step was crucial for focusing the LLMs on essential medical information, reducing prompt size, and lowering computational costs.
Eleven different LLMs were evaluated, categorized into reasoning models (e.g., DeepSeek Reasoner, Gemini 2.5 Pro, GPT o3 Mini) and non-reasoning models (e.g., Llama 4 Scout, GPT 4o, DeepSeek Chat). A standardized prompt was engineered to simulate a clinical coder, asking the models to identify primary and secondary ICD-10 codes in a specific format. Model performance was assessed using the F1 score across three ICD-10 levels of granularity: Level 3 (general), Level 4, and Level 5 (most specific), for both primary and all diagnoses classification tasks.
Key Findings
The study revealed that while LLMs show potential, none achieved F1 scores above 57% for any classification task, indicating they are not yet ready for autonomous deployment. However, reasoning models generally outperformed non-reasoning models, suggesting that explicit reasoning processes enhance classification accuracy. The Gemini 2.5 Pro model consistently demonstrated the highest performance across most tasks and ICD-10 levels.
A significant observation was the decline in performance as ICD code specificity increased. Models performed best at Level 3 (more general codes) and struggled significantly at Level 5 (most specific codes). Certain codes, like I25 (chronic ischemic heart disease), were classified more accurately, while others, such as Y92 or Z51, consistently yielded near-zero F1 scores, highlighting inherent difficulties with specific diagnostic categories.
The “all diagnoses” classification task, which allows for multiple ICD-10 codes including primary and secondary, saw improved performance compared to the “primary diagnosis” task, which demands a single, most accurate code. This suggests LLMs are better at identifying a broader range of relevant codes rather than pinpointing a single primary one with high precision.
Also Read:
- Enhancing French Electronic Health Records with AI for Social Determinants of Health
- AI Models Streamline Healthcare Documentation with New Clinical Datasets
Implications and Future Directions
The findings suggest that LLMs, particularly those with structured reasoning capabilities, can support clinical coders by augmenting their work, but they are not yet suitable for full automation. Their effectiveness is limited by the granularity and variability of ICD-10 codes. The novel integration of cTAKES for data preprocessing was a key contribution, enhancing the contextual understanding for LLMs.
Future research should focus on developing hybrid models that combine LLMs with rule-based logic, fine-tuning models on domain-specific data, and integrating structured clinical data like lab results. Ethical considerations, including transparency, auditability, and privacy compliance, remain crucial for the safe integration of AI into healthcare workflows.
In conclusion, while LLMs with reasoning capabilities show promise for semi-automated clinical coding, especially for higher-level classifications, significant advancements are needed to achieve the precision required for autonomous deployment in complex medical coding scenarios.
