TLDR: This research explores fine-tuning PEGASUS and PEGASUS-X deep learning models for abstractive summarization of radiological reports using limited medical data. It highlights challenges like overfitting and “catastrophic forgetting,” emphasizing the critical need to monitor training history and showing that larger models aren’t always superior when data is scarce and pre-training objectives differ.
The rapid advancements in artificial intelligence have opened new avenues for automating complex tasks, but abstractive summarization, especially in sensitive and data-restricted fields like medicine, continues to pose significant challenges. With the ever-increasing volume of medical imaging, there’s a growing demand for automated tools that can efficiently distill clinically relevant information from intricate textual data, such as radiological reports.
A recent empirical study, titled Deep learning and abstractive summarisation for radiological reports: an empirical study for adapting the PEGASUS models’ family with scarce data, delves into adapting non-domain-specific abstractive summarization models for this critical medical application. The research, conducted by Claudio Benzoni, Martina Langhals, Martin Boeker, Luise Modersohn, and M´ at´ e E. Maros, specifically investigates the fine-tuning process of the PEGASUS and PEGASUS-X model families, providing valuable insights for practitioners on how to navigate the pitfalls of over- and underfitting when working with limited data.
Unlike the large language models (LLMs) that often require substantial computational resources, encoder-decoder architectures like PEGASUS and PEGASUS-X are particularly efficient and well-suited for sequence-to-sequence tasks such as summarization. The study utilized a publicly available dataset of medium-sized radiological reports, comprising 3996 X-ray reports, to adapt these models. The goal was to summarize longer ‘findings’ sections into shorter ‘impressions,’ while preserving the semantic meaning.
The researchers comprehensively evaluated two different checkpoints for each model, varying the sizes of the training data (10%, 50%, and 100% of the original dataset). They meticulously monitored the models’ performance using both traditional lexical metrics (like ROUGE and BLEU) and more advanced semantic metrics (such as METEOR and BERTScore) throughout the training history.
Key Findings from PEGASUS Models
The PEGASUS models exhibited distinct behaviors depending on the training data size. With the smallest training set (10%), both checkpoints initially showed a sharp increase in performance, followed by a significant degradation. This phenomenon is a classic sign of overfitting, where the model learns to reproduce the training data too well but fails to generalize to unseen data.
More interestingly, when trained with 50% and 100% of the data, the PEGASUS models displayed three phases: an early peak in performance, a subsequent drop (termed the ‘forgetting phase’), and finally, a stationary phase where the models regained and surpassed their early peak performance, acquiring better generalization abilities. This behavior is conjectured to be a manifestation of ‘epoch-wise double-descent,’ a phenomenon observed in deep neural networks where performance can initially worsen before improving again with more training.
The study also found that the ‘xsum’ checkpoint of PEGASUS, which was pre-trained on a dataset favoring concise summaries, performed better than the ‘large’ checkpoint, aligning well with the typically short summaries found in chest X-ray reports.
Insights from PEGASUS-X Models
The PEGASUS-X models, which differ in parameter count rather than pre-training dataset, presented a different set of observations. The smaller ‘base’ checkpoint showed a nearly monotonous increase in metrics, with performance plateauing faster as the training data size increased. This indicates a more stable learning process.
However, the larger ‘large’ checkpoint of PEGASUS-X exhibited puzzling behavior. Its metric scores were consistently lower than the smaller checkpoint, and its training history was erratic, characterized by jagged performance curves. The model struggled to understand the summarization task, often producing outputs that were a mix of its pre-training data and the fine-tuning data. This suggests that simply using a larger model with more parameters does not guarantee improved performance, especially when the model’s pre-training objectives differ significantly from the fine-tuning task and the available data is scarce.
Also Read:
- Enhancing AI Trustworthiness: A New Approach to Curb Reward Hacking in Medical LLMs
- New Multimodal AI Framework for Lung Nodule Diagnosis: Insights and Challenges
Conclusion and Practical Advice
This research underscores the inherent challenges of fine-tuning large, expressive deep learning models in clinical environments where labeled training data is often limited. The study highlights that naive fine-tuning on datasets with limited variability can lead to ‘catastrophic forgetting’ and complex performance patterns like ‘peak-and-drop recovery.’
A crucial takeaway for clinicians and practitioners is the importance of carefully monitoring the entire training history of these models. Aggressive early-stopping techniques, which might halt training at an initial peak, could prevent the model from reaching its optimal performance in a later generalization phase. Furthermore, the study cautions against the assumption that larger models are always better; in scenarios with scarce data and differing pre-training objectives, a smaller, more focused model might yield superior results.
While acknowledging limitations such as the dataset size and focus on a single model family, this work lays essential groundwork for developing more robust fine-tuning strategies for summarization models in specialized domains, paving the way for more reliable AI applications in healthcare.
