TLDR: A study evaluated OpenAI’s gpt-4o model for zero-shot classification of pneumonia from chest X-ray images. Using a balanced test set of 400 images, researchers tested four prompt designs. The most effective prompt, which included visual features, achieved 74% accuracy, while prompts requiring detailed reasoning performed worse. The findings suggest that while multimodal models like gpt-4o have potential in medical imaging, their diagnostic reliability is currently limited and requires further development for clinical use.
Pneumonia, an infection causing inflammation in the lung’s air sacs, remains a significant global health challenge. It can be life-threatening, especially for infants, the elderly, and those with weakened immune systems. The disease leads to millions of emergency room visits and tens of thousands of deaths annually in the United States, and tragically, over 800,000 deaths in children under five worldwide in 2017. These high mortality rates are often linked to delayed diagnoses, lack of equipment in low-resource areas, and the difficulty in distinguishing pneumonia from other respiratory conditions. Current diagnostic methods typically involve a doctor’s examination, blood tests, chest X-rays, pulse oximetry, and sputum tests.
The Promise of AI in Diagnosis
Given the critical need for faster, more efficient, and cost-effective diagnostic methods, artificial intelligence (AI) offers a promising avenue. Misdiagnosis of pneumonia can have severe consequences, with studies indicating that up to twelve percent of all pneumonia diagnoses among hospitalized adults might be inappropriate. AI has the potential to provide more consistent and rapid interpretations of medical scans and X-rays, thereby improving patient outcomes.
This is where zero-shot learning (ZSL) comes into play. ZSL is a machine learning paradigm where a model can recognize and classify new categories it has not been explicitly trained on. Instead of relying on vast labeled datasets, ZSL leverages a model’s ability to connect visual inputs with semantic or textual descriptions learned during its pre-training. In medical imaging, this means an AI model, such as a large vision-language model, can analyze radiographs and infer diagnoses based on its general understanding, rather than needing direct examples of every specific condition.
The Role of Prompts in AI Interpretation
In zero-shot image classification using advanced multimodal models, prompts are crucial. They act as the bridge between human instructions and the model’s behavior, defining the task, providing linguistic context, and specifying the desired output. Effective prompt design helps align the model’s broad visual-linguistic knowledge with the specific task, highlighting relevant features and reducing ambiguity. Prompts can range from simple classification requests to more complex instructions that guide the model to explain its reasoning or use specialized medical terminology. By carefully structuring these instructions, zero-shot models can apply their learned knowledge to new domains without explicit supervision.
Evaluating ChatGPT’s Performance
A recent study, titled Evaluating ChatGPT’s Performance in Classifying Pneumonia from Chest X-Ray Images, investigated the ability of OpenAI’s gpt-4o model to classify chest X-ray images as either NORMAL or PNEUMONIA in a zero-shot setting, meaning without any prior fine-tuning for this specific task. The researchers used a balanced test set of 400 images (200 from each class) from the publicly available Chest Radiographic Images (Pneumonia) dataset on Kaggle. The images were sourced from pediatric patients aged one to five years.
The study tested four distinct prompt designs, ranging from minimal instructions to detailed, reasoning-based prompts. Each interaction involved a system prompt, which set the model’s role (e.g., a medical image labeler outputting JSON), and a user prompt, which provided specific instructions for the image. The four prompt types were:
- Prompt 1: Minimal Output (No Features) – The simplest, asking only for a label and confidence.
- Prompt 2: Including Features in Output – Asked the model to list short visual features it noticed.
- Prompt 3: Features + Concise Reasoning – Added a short, one- or two-sentence justification.
- Prompt 4: Features + Step-by-Step Reasoning – Required a more detailed, step-by-step explanation of its reasoning.
Also Read:
- CXRAgent: A New AI Approach for Reliable Chest X-Ray Interpretation
- Boosting Trust in AI-Generated Liver MRI Reports: A New Framework for Prompt Optimization and Credibility Assessment
Key Findings and Limitations
The results showed that concise, feature-focused prompts achieved the highest classification accuracy. Specifically, Prompt 2, which asked the model to include short visual features, achieved the best accuracy of 74%. Interestingly, prompts that required longer or step-by-step reasoning (Prompts 3 and 4) did not improve performance; in fact, they slightly reduced accuracy. This suggests that while reasoning-oriented large language models are powerful for text-based tasks, they may not yet be optimized for visual diagnostic reasoning, especially in medical contexts where brief, focused reasoning appears more effective.
In conclusion, this study highlights that while ChatGPT (gpt-4o) demonstrates emerging potential for medical image interpretation, its diagnostic reliability remains limited. An overall accuracy of 74% indicates that the model still struggles to consistently recognize subtle medical image features. Continued advances in visual reasoning, specialized medical fine-tuning, and integration with domain-specific datasets are essential before such models can be safely and reliably applied in clinical practice.
