TLDR: MedRepBench is a new, comprehensive benchmark for evaluating how well AI models, especially Vision-Language Models (VLMs), interpret structured information from real-world Chinese medical reports. It includes 1,900 de-identified reports and features both objective (field-level recall) and automated subjective (LLM-based interpretability) evaluation protocols. The research shows that while OCR-assisted methods perform well, end-to-end VLMs have significant potential, particularly when optimized with reinforcement learning, which led to a 6% recall gain in a mid-scale VLM, outperforming larger models.
Medical report interpretation is a vital component of modern healthcare, helping both patients understand their health information and clinical systems share data efficiently. While advanced AI models like Vision-Language Models (VLMs) and Large Language Models (LLMs) have shown promise in understanding documents, there hasn’t been a standard way to measure how well they interpret structured information in medical reports.
To address this gap, researchers have introduced MedRepBench, a new comprehensive benchmark designed specifically for evaluating the structured interpretation quality of medical reports. This benchmark is built from 1,900 real-world, de-identified Chinese medical reports, covering a wide range of departments, patient demographics, and acquisition formats (like photos, screenshots, and electronic documents). The primary goal of MedRepBench is to assess end-to-end VLMs in understanding structured medical content directly from images.
Understanding MedRepBench’s Approach
MedRepBench offers two main evaluation methods. First, an objective evaluation measures the field-level recall of structured clinical items. This means it checks how accurately models can extract specific pieces of information, such as a test name, its value, unit, reference range, and whether it’s abnormal. Second, an automated subjective evaluation uses a powerful LLM as a scoring agent to assess the factuality, interpretability, and reasoning quality of the generated explanations. This dual approach provides a comprehensive view of a model’s performance.
The dataset itself is diverse, including both examination and laboratory reports captured through various methods, reflecting the real-world variability in medical documentation across hundreds of hospitals. This heterogeneity is crucial for testing the robustness and generalizability of AI models.
Key Findings and Reinforcement Learning
The research conducted with MedRepBench revealed several important insights. While OCR (Optical Character Recognition) combined with LLMs showed strong performance, it often suffered from ‘layout-blindness’ and latency issues, as it loses visual and spatial information. This highlights the need for fully vision-based report understanding.
Interestingly, the study also demonstrated the power of reinforcement learning (RL) in improving VLM performance. By designing a reward function based on field-level recall and applying Group Relative Policy Optimization (GRPO), a mid-scale VLM achieved up to a 6% gain in recall. This RL-optimized model, called Ours-GRPO, even outperformed several larger-scale models in structured field extraction without OCR input, showcasing the potential of targeted optimization.
The benchmark also evaluated leading open-source VLMs and LLMs under both end-to-end (image-only) and OCR-assisted settings. Results showed that performance generally scales with model capacity, but alignment strategies and training data play crucial roles. The gap in performance between models with and without OCR input indicates significant room for improvement in visual-text alignment and structured reasoning for VLMs.
Also Read:
- Bridging the Language Divide: New Benchmark Reveals LVLM Struggles with Numerical and Cross-Lingual Visual Reasoning
- Advanced Medical AI: Hierarchical Vision-Language Learning for Out-of-Distribution Disease Detection
Subjective Evaluation and Human Agreement
Beyond just extracting data, MedRepBench assesses how well models can generate human-readable and clinically appropriate explanations. Using DeepSeek-R1 as an evaluator, models were scored on factual accuracy, reasoning validity, and ethical compliance. Larger VLMs generally achieved higher interpretability scores, but the RL-optimized model also performed strongly.
To ensure the reliability of the LLM-based subjective evaluation, a consistency study was conducted against expert human judgments. The agreement was strong, with an accuracy of 88.3% and a Cohen’s kappa coefficient of 0.82, indicating that LLM-based evaluation can be a viable and scalable method for assessing medical interpretation tasks.
In conclusion, MedRepBench provides a much-needed standardized benchmark for evaluating AI models in the complex task of medical report interpretation. It emphasizes the importance of end-to-end, layout-aware understanding and demonstrates how targeted reinforcement learning can significantly enhance model performance. The dataset and evaluation toolkit will be made publicly available, fostering further research and development in this critical area of healthcare AI. You can read the full research paper here.
