TLDR: PARROT is the largest open-access, multilingual dataset of 2658 fictional radiology reports from 76 radiologists across 21 countries and 13 languages. It aims to overcome privacy and linguistic barriers in developing AI for radiology by providing diverse, expert-authored content for natural language processing applications. A study showed that even radiologists found it hard to distinguish these human-authored reports from AI-generated ones, emphasizing the value of the human-authored content.
In the rapidly evolving field of artificial intelligence in healthcare, a significant challenge has been the lack of diverse, openly accessible data, especially for radiology reports across different languages. Most existing datasets are primarily in English, limiting the development and application of AI tools globally. To address this critical gap, a new initiative called PARROT (Polyglottal Annotated Radiology Reports for Open Testing) has introduced the largest open-access multilingual radiology reports dataset to date.
The PARROT dataset is the brainchild of a collaborative effort led by Bastien Le Guellec and a large international team of radiologists and AI researchers. Their work, detailed in their research paper, aims to provide a robust resource for testing natural language processing (NLP) applications in radiology, transcending linguistic, geographic, and clinical boundaries without the usual privacy constraints associated with real patient data. You can find more details about their work here: PARROT Research Paper.
What is PARROT and Why is it Important?
Radiology reports are vital for communicating complex visual findings from medical images to clinicians, directly influencing patient care. While large language models (LLMs) have shown immense potential in enhancing radiology workflows—from structuring free-text findings to detecting inconsistencies—their widespread deployment is hindered by their English-language centricity. Existing large datasets like MIMIC-IV and MIMIC-CXR contain thousands of English reports, but privacy regulations often restrict the sharing of non-English clinical data.
PARROT tackles this by comprising entirely fictional, yet realistic, radiology reports. This innovative approach allows for unrestricted sharing, modification, and augmentation under a CC-BY-NC-SA 4.0 license, enabling collaborative research across institutional and national borders without privacy concerns. It also addresses the issue of diverse reporting practices worldwide, which vary not just in language but also in style, format, and terminology.
How Was the Dataset Created?
The PARROT initiative ran from May to September 2024. Radiologists from around the globe were invited to contribute at least 20 fictional radiology reports, adhering to their typical reporting styles for their respective languages and regions. Each submission included metadata such as anatomical region, imaging modality (like CT, MRI, ultrasound, X-ray), brief clinical context, and, for non-English reports, an English translation. All reports were also assigned ICD-10 codes, which classify diseases and health problems.
Contributors were encouraged to create plausible but non-specific clinical scenarios, including typical incidental findings and normal anatomical variants, to ensure the dataset reflects a realistic distribution of findings while capturing authentic reporting styles.
Key Features of the PARROT Dataset
The dataset is impressive in its scale and diversity:
- It contains 2658 fictional radiology reports.
- Contributions came from 76 authors across 21 countries and 4 continents.
- It encompasses 13 different languages, with Polish, German, Italian, and French being the most prevalent. Notably, French reports originated from multiple countries, highlighting regional variations within the same language.
- Reports cover multiple imaging modalities: CT (36.1%), MRI (22.8%), conventional radiography (19.0%), and ultrasound (16.8%).
- The most prevalent anatomical regions covered are chest (19.9%), abdomen (18.6%), head (17.3%), and pelvis (14.1%).
- The median report length varies significantly by language, from Afrikaans (36.5 words) to Turkish (382 words).
- The dataset includes pathologies across all major ICD-10 chapters, with a predominance in circulatory, respiratory, digestive, and musculoskeletal system diseases.
Distinguishing Human from AI-Generated Reports
To validate the authenticity of the human-authored reports, a differentiation study was conducted with 154 participants, including radiologists, other healthcare professionals, and non-healthcare professionals. Participants were asked to identify whether reports were human-authored or AI-generated (using GPT-o1). Overall, participants achieved only 53.9% accuracy, just slightly above chance level.
Interestingly, radiologists performed significantly better (56.9%) than non-healthcare professionals (49.7%) and other healthcare professionals (48.3%). This suggests that domain expertise provides some advantage in discerning report authenticity, highlighting the value of professionally authored content in the PARROT dataset compared to purely synthetic AI-generated text, which might contain subtle errors unnoticed by non-experts.
Also Read:
- Adaptive Group Alignment: A New Framework for Understanding Medical Images and Reports
- Unmasking AI-Generated Code: A New Dataset for Detection Across Languages and Scenarios
Conclusion and Future Outlook
PARROT stands as a crucial resource for the development and validation of NLP applications in radiology. By offering fictional yet radiologist-authored reports across multiple languages, it provides a valuable benchmark for creating and testing AI tools that can function effectively across diverse healthcare systems and languages, overcoming the significant linguistic and accessibility constraints that have previously hindered progress in this field.
While the dataset is a major step forward, the authors acknowledge some limitations, such as a geographical imbalance (Europe predominates) and the lack of links to real patient imaging and outcomes. However, the open-access nature and ongoing development of PARROT promise to foster more inclusive and globally relevant AI solutions in radiology.
