TLDR: A study successfully instruction-tuned open-weight Large Language Models (LLMs) using public German medical catalogues (ICD-10-GM, ICD-O-3, OPS) to significantly improve the accuracy of coding German tumor diagnoses. Fine-tuning boosted ICD-10-GM coding accuracy from 1.4-24% to 41-58% and ICD-O-3 topography coding accuracy to 22-40%, while eliminating malformed outputs and achieving 99% tumor diagnosis recognition. This highlights the untapped potential of public data for enhancing LLMs in medical documentation.
Accurate coding of tumor diagnoses is a critical component of cancer documentation in Germany. This process relies on established classification systems like ICD-10-GM and ICD-O-3. However, automating this complex, text-based task, especially in non-English languages like German, has presented significant challenges for large language models (LLMs).
Addressing the Coding Challenge with AI
A recent study, “Unlocking Public Catalogues: Instruction-Tuning LLMs for ICD Coding of German Tumor Diagnoses,” explores how instruction-based fine-tuning using publicly available data can significantly improve the coding accuracy of open-weight LLMs for German tumor diagnosis texts. The research was conducted by Stefan Lenz, Lakisha Ortiz Rosario, Georg Vollmar, Arsenij Ustjanzew, Fatma Alickovic, Thomas Kindler, and Torsten Panholzer from institutions including the University Medical Center of the Johannes Gutenberg University Mainz.
The core idea behind this research is to leverage existing public medical catalogues to create a comprehensive training dataset. Over 500,000 question-answer pairs were generated from the German versions of the ICD-10-GM, ICD-O-3, and OPS catalogues. These pairs covered various tasks, including coding tumor diagnoses according to ICD-10 and ICD-O, and distinguishing tumor diagnoses from other diseases or medical procedures. The instructions were phrased in German, and answers were kept short to train models for concise responses.
Methodology and Evaluation
Eight open-weight LLMs from the Qwen, Llama, and Mistral families, ranging from 7 billion to 70 billion parameters, were fine-tuned using a parameter-efficient technique called Low-Rank Adaptation (LoRA). The models were evaluated using a real-world dataset of 2024 unique German tumor diagnosis descriptions from the University Cancer Center at the University Medical Center Mainz, specifically from 2023 and 2024 to ensure relevance with current catalogue versions.
A crucial aspect of the study was a systematic data quality assessment. Researchers found that perfect model performance (100% accuracy) couldn’t be expected due to incomplete information in the short free-text diagnosis descriptions. For instance, missing tumor behavior (benign/malign/uncertain) or anatomical localization could prevent unambiguous code assignment. The analysis estimated an upper bound of approximately 70% for complete ICD-10-GM code prediction and 65% for exact ICD-O localization, providing a realistic benchmark for model performance.
Significant Improvements in Coding Accuracy
The results demonstrated substantial improvements after fine-tuning. ICD-10-GM coding accuracy for the models rose dramatically from a baseline of 1.4-24% to 41-58%. Partial accuracy (correct three-character codes) also increased from 31-74% to 73-83%. Similarly, ICD-O-3 topography coding accuracy improved, reaching 22-40% for exact accuracy and 56-67% for partial accuracy after fine-tuning, although it started and remained lower than ICD-10 coding.
A notable achievement was the reduction of malformed code outputs to 0% for all models, indicating that the fine-tuning successfully taught the models to produce valid code formats. Tumor diagnosis recognition, a simpler task, reached an impressive 99% accuracy, even for base models that initially struggled in some cases. The study also observed a positive correlation between model size and accuracy, with larger models generally performing better, though fine-tuning narrowed the performance gap between smaller and larger models.
Interestingly, the “thinking mode” in Qwen3 models, which involves generating intermediate reasoning steps, did not improve accuracy and significantly increased response times (over 100-fold slower). Fine-tuned models, conversely, responded even faster than base models, aligning with the training objective for concise answers.
Also Read:
- General Language Models Show Promise in Extracting Medical Orders from Doctor-Patient Conversations
- Mapping Medical Knowledge Within Large Language Models: A Deep Dive into AI Interpretability
Future Directions and Impact
The findings underscore the significant, often underused, potential of public catalogues to create instruction datasets that enhance LLMs for medical documentation tasks. While limitations exist, such as the inherent incompleteness of short diagnosis texts, the study highlights that providing richer contextual data in real-world applications could further boost performance beyond the evaluated bounds.
The researchers suggest that future progress could come from more diverse instructions addressing the varied challenges of medical documentation, especially for ICD-O coding, where performance lagged due to weaker prior knowledge and less diverse training coverage. Combining generative models with embedding vector approaches, such as Retrieval-Augmented Generation (RAG), is also identified as a promising direction for further improvement.
This research demonstrates a transferable approach for improving medical coding capabilities of LLMs, not only for German but potentially for other languages and similar classification systems. The complete training dataset and the best-performing checkpoints of the fine-tuned models are publicly available on Hugging Face, fostering further research and development in this crucial area. You can read the full research paper here.
