TLDR: A study benchmarked GPT-5 in radiation oncology, finding it significantly outperformed previous models (GPT-4, GPT-3.5) on a multiple-choice exam (92.8% accuracy) and showed strong performance in generating treatment recommendations for real-world cases. While hallucinations were rare, errors occurred in complex scenarios requiring precise trial knowledge. The research concludes that GPT-5 is a valuable assistant for tasks like education and drafting treatment plans, but human expert oversight remains crucial for clinical implementation.
Large language models, or LLMs, have rapidly advanced, showing immense promise in various scientific and clinical fields. Among these, the medical domain, particularly oncology, stands to benefit significantly from AI-driven decision support and educational tools. A recent study, available as a preprint, delves into the capabilities of GPT-5, a novel LLM system specifically marketed for oncology use, within the specialized field of radiation oncology. This research provides a comprehensive benchmark of GPT-5’s performance, highlighting both its measurable gains and the persistent need for expert human oversight. You can read the full paper here: Benchmarking GPT-5 in Radiation Oncology.
The study, conducted by a team of researchers including Ugur Dinc, Jibak Sarkar, and Florian Putz, aimed to assess GPT-5’s proficiency using two distinct benchmarks. The first was the American College of Radiology Radiation Oncology In-Training Examination (TXIT, 2021), a standardized test comprising 300 multiple-choice questions. The second, and perhaps more clinically relevant, involved a curated set of 60 authentic radiation oncologic patient cases, or vignettes, covering a wide range of disease sites and treatment indications. For these vignettes, GPT-5 was tasked with generating structured therapeutic plans and concise summaries, which were then independently rated by four board-certified radiation oncologists for correctness, comprehensiveness, and the presence of ‘hallucinations’ (factually incorrect or nonsensical information).
The results on the TXIT benchmark were striking. GPT-5 achieved an impressive mean accuracy of 92.8%, significantly outperforming its predecessors, GPT-4 (78.8%) and GPT-3.5 (62.1%). This indicates a substantial leap in the model’s ability to recall and apply knowledge in a standardized test setting. The improvements were particularly noticeable in areas like dose specification and diagnosis, suggesting enhanced domain-specific understanding.
In the more complex real-world vignette evaluation, GPT-5’s treatment recommendations were generally well-received. They were rated highly for correctness, with a mean score of 3.24 out of 4, and even higher for comprehensiveness, averaging 3.59 out of 4. A crucial finding was the infrequency of hallucinations; only 10% of individual ratings identified them, and no single case was flagged by a majority of experts. This suggests that GPT-5 is largely reliable in generating factually sound information. However, the study also noted low inter-rater reliability among the human oncologists, reflecting the inherent variability and subjectivity in clinical judgment, even among experts.
Despite these strong performances, the study identified specific areas where GPT-5 still faces challenges. Errors tended to cluster in complex scenarios that demanded precise knowledge of clinical trials, nuanced clinical adaptation, or intricate multi-modality treatment sequencing. For instance, cases involving rectal/anal cancers, certain lung cancer scenarios, and specific breast or metastatic disease subgroups showed lower correctness scores and higher variability. Examples of limitations included recommending overtreatment for low-risk prostate cancer, omitting crucial biomarker analysis in rectal cancer, or proposing non-guideline-concordant dosing schemes for systemic therapies.
The researchers emphasized that GPT-5’s advancements, particularly its explicit positioning as a ‘reasoning model’ designed to generate structured rationales, represent a qualitative step forward. This capability allows it to synthesize case-relevant rationales, moving closer to providing the deliberative support needed in settings like tumor boards. However, the consistent message throughout the paper is the indispensable role of human oversight. While GPT-5 can generate coherent and comprehensive drafts, these recommendations require rigorous expert review before clinical implementation, especially in high-stakes oncology settings.
Also Read:
- GPT-5 Unleashes ‘Vibe Coding’ and Agentic AI to Reshape Software Development Landscape
- MEDLEY: Harnessing AI’s Imperfections for Smarter Medical Decisions
In conclusion, GPT-5 emerges as a powerful augmentative assistant for radiation oncology. Its strengths lie in educational applications, pre-board preparation, and generating initial drafts for tumor board discussions. The study underscores that while AI models like GPT-5 offer significant gains in efficiency and knowledge synthesis, they are not yet ready for autonomous decision-making. Human verification and the integration of evidence retrieval systems remain essential safeguards to ensure patient safety and maintain accountability in clinical practice.
