TLDR: Researchers have created CPC-Bench, a new benchmark using over 7,000 historical medical cases from the New England Journal of Medicine, to thoroughly evaluate AI in medical reasoning. They also developed “Dr. CaBot,” an AI capable of generating expert-level differential diagnoses and video presentations. The study found that advanced AI models now surpass human physicians in complex text-based diagnoses and can convincingly emulate expert medical presentations, though they still face challenges in image interpretation and literature retrieval.
For over a century, the New England Journal of Medicine Clinicopathological Conferences (CPCs) have served as a rigorous test for expert physicians’ reasoning and, more recently, for artificial intelligence (AI). These challenging clinical cases, published continuously since 1923, offer a unique window into the evolution of disease classification and clinical reasoning. However, past AI evaluations often focused solely on final diagnoses, overlooking the complex reasoning and presentation skills expected of human experts.
A groundbreaking new research paper, titled “Advancing Medical Artificial Intelligence Using a Century of Cases,” introduces a comprehensive approach to evaluating medical AI. Authored by Thomas A. Buckley, Riccardo Conci, Peter G. Brodeur, Jason Gusdorf, Sourik Beltrán, Bita Behrouzi, Byron Crowe, Jacob Dockterman, Muzzammil Muhammad, Sarah Ohnigian, Andrew Sanchez, James A. Diao, Aashna P. Shah, Daniel Restrepo, Eric S. Rosenberg, Andrew S. Lea, Marinka Zitnik, Scott H. Podolsky, Zahir Kanjee, Raja-Elie E. Abdulnour, Jacob M. Koshy, Adam Rodman, and Arjun K. Manrai, this study addresses the limitations of previous evaluations by creating a new benchmark and an AI expert discussant.
Introducing CPC-Bench: A New Standard for Medical AI Evaluation
The researchers developed CPC-Bench, a physician-validated benchmark that leverages 7102 CPCs (from 1923-2025) and 1021 Image Challenges (from 2006-2025). This extensive dataset underwent physician annotation and automated processing to create a benchmark spanning 10 text-based and multimodal tasks. These tasks include generating differential diagnoses, creating testing plans, literature searching, diagnostic reasoning at various ‘touchpoints’ in a patient’s course, question-answering, and visual interpretation tasks.
Meet Dr. CaBot: The AI Expert Discussant
In parallel with CPC-Bench, the team developed “Dr. CaBot,” an AI discussant designed to mimic human expert physicians. Dr. CaBot can produce written and slide-based video presentations using only the initial case presentation, just like a human expert. It achieves this by identifying similar historical cases and iteratively querying a vast clinical literature database. The system was refined based on physician feedback to closely emulate the style and format of published CPC differentials.
Key Findings: AI’s Strengths and Remaining Challenges
The evaluation of leading large language models (LLMs) against CPC-Bench revealed impressive capabilities and highlighted areas for improvement:
- Differential Diagnosis: The o3 model demonstrated remarkable performance, ranking the final diagnosis first in 60% of cases and within the top ten in 84% of cases, surpassing a baseline of 20 physicians. Its next-test selection accuracy reached an impressive 98%.
- Diagnostic Progression: Models showed stepwise gains in diagnostic accuracy as more clinical information (touchpoints) was provided throughout a case.
- Literature Search: While models like Gemini 2.5 Pro performed better (49% top-10 accuracy) than o3 (32%) in recalling citations from memory, literature retrieval remains a weaker area compared to diagnostic generation. Retrieval-augmented generation (RAG) showed promise in improving performance.
- Question-Answering: Frontier models like o3 were highly proficient, providing correct answers in 88% of clinical knowledge questions.
- Image Interpretation: This area presented more challenges. On the NEJM Image Challenge, Gemini 2.5 Pro and o3 achieved 84% and 82% accuracy, respectively. However, for image-only diagnoses, accuracy dropped to 67%. Performance varied by specialty, with dermatology images being easier than radiology images.
- Dr. CaBot’s Emulation: In blinded comparisons, physicians struggled to distinguish between CaBot-generated and human expert-generated differential diagnoses, misclassifying the source in 74% of trials. Physicians even rated Dr. CaBot more favorably across several quality dimensions, including overall quality and diagnostic justification.
- Historical Performance: When benchmarked against historical cases from 1923 to 2025, o3 performed on par with human expert discussants after the 1960s and even outperformed them during the 1940s and 1950s.
Also Read:
- Assessing AI’s Clinical Acumen: Introducing PsychiatryBench for Language Models in Mental Health
- Enhancing Clinical Reasoning in AI: A Compartmentalized Approach for Language Models
The Future of Medical AI
The study concludes that LLMs now rival human experts in complex text-based clinical reasoning and can convincingly emulate expert medical presentations. However, challenges persist in literature retrieval, image interpretation, and multimodal integration. The researchers emphasize that advancements in model scale, pretraining, and architectures optimized for reasoning are the primary drivers of progress in clinical AI, rather than specialized fine-tuning or complex prompting techniques.
To foster further research and transparently track progress, the researchers are publicly releasing CPC-Bench and Dr. CaBot. These resources extend Dr. Richard Cabot’s original vision of the CPC as a teaching tool and a benchmark for clinical reasoning, now for both humans and machines. You can explore the full research paper here: Advancing Medical Artificial Intelligence Using a Century of Cases.
