TLDR: The TREC PLABA track evaluated Large Language Models (LLMs) for simplifying biomedical abstracts. Top models achieved high factual accuracy but struggled with simplicity and brevity. Automatic evaluation metrics showed poor correlation with human judgments, highlighting their insufficiency. The track also revealed challenges with identifying complex terms and the persistent risk of LLM hallucinations, emphasizing the need for careful deployment and improved automated fact-checking tools.
Understanding complex medical research can be a significant challenge for patients and caregivers, often creating a barrier to accessing vital health information. While the internet offers a wealth of data, much of it is written in highly technical language, making it difficult for the general public to comprehend. This “language barrier” can hinder health outcomes and prevent individuals from staying informed about the latest medical advancements.
To address this critical issue, the Text REtrieval Conference (TREC) introduced a new shared task: the Plain Language Adaptation of Biomedical Abstracts (PLABA) track. Hosted in 2023 and 2024, the PLABA track aimed to stimulate research and provide a high-quality evaluation of systems designed to adapt professional biomedical literature into plain language, making it more accessible to a broader audience.
The PLABA Track: Objectives and Methods
The primary objective of the PLABA track was to rigorously evaluate the potential of advanced language models in simplifying biomedical text. Given the high potential for harm if medical information is misinterpreted, a thorough evaluation was deemed essential.
The track featured two main tasks:
- Task 1: Rewriting Abstracts – This task challenged participants to rewrite entire biomedical abstracts sentence-by-sentence for a general audience. The goal was to preserve as much information as possible while making it understandable, similar to ‘translating’ from expert to plain language.
- Task 2: Identifying and Replacing Complex Terms – Introduced in the second year, this task focused on identifying difficult terms within abstracts and providing appropriate replacements or explanations. Five types of simplification were defined: Substitute (e.g., “heart attack” for “myocardial infarction”), Explain (providing a definition), Generalize (using a broader term), Exemplify (giving examples), and Omit (removing unnecessary terms).
For evaluation, Task 1 submissions underwent extensive manual review by biomedical experts, assessing simplicity, accuracy, completeness, and brevity. Automatic metrics like SARI, BLEU, ROUGE, and BERTScore were also used, though their correlation with human judgments was a key finding. Task 2 involved automatic evaluation for identification and classification, with manual evaluation for the generated simplifications.
Key Findings and Lessons Learned
The PLABA track attracted diverse teams from twelve countries, employing a wide array of models including BERT, BART, T5, GPT, Llama, Gemini, and Mistral. The results offered valuable insights into the current capabilities and limitations of large language models in this domain.
In Task 1, top-performing models demonstrated impressive factual accuracy and completeness, sometimes rivaling human performance in these aspects. However, they consistently fell short in achieving human levels of simplicity and brevity. This suggests that while models can retain information, making it truly concise and easy to read remains a significant challenge. A crucial observation was the poor correlation between automatic, reference-based metrics (like SARI) and manual judgments, indicating that current automated evaluation tools may not fully capture the nuances of plain language adaptation.
For Task 2, systems struggled with accurately identifying difficult terms and classifying the best way to replace them. However, when it came to generating replacements, LLM-based systems performed well in terms of accuracy, completeness, and simplicity, though brevity remained an issue.
Challenges and Future Directions
The track highlighted several areas for future work. One significant challenge is the issue of “hallucination,” where models generate plausible but factually incorrect information. While top systems were generally highly factual, even rare instances of fabrication in the biomedical domain carry a high potential for harm. This underscores the need for more robust methods to detect and mitigate hallucinations and to automatically assess factuality without relying on simple word similarity.
Another area for improvement is the development of better automatic benchmarking tools that align more closely with human judgments of simplicity and factual content. The track also explored sentence-level adaptation while accounting for document context, a complex challenge that requires further refinement in evaluation methods, especially regarding sentence merging and transposition.
Also Read:
- KROMA: Enhancing Ontology Matching with Context-Aware Language Models
- Building Reliable AI: Bridging Large Language Models and Expert Systems
Conclusion
The PLABA track demonstrated the significant promise of Large Language Models in adapting complex biomedical literature for the general public, potentially enhancing healthcare engagement and literacy. While the best systems achieved near-human levels of factual accuracy and simplification, caution is advised due to lingering deficiencies in brevity and the potential for harmful hallucinations. The lessons learned from PLABA will undoubtedly inform and inspire future research in consumer-focused biomedical text generation, guiding a “progressive caution” approach to deployment. You can read the full research paper for more details: Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track.
