TLDR: BenchReAD is a new, comprehensive benchmark for retinal anomaly detection, addressing limitations of previous benchmarks by offering diverse datasets (fundus and OCT) with both seen and unseen anomalies, and systematically evaluating methods across different supervision levels. While fully supervised methods like DRA perform well, they struggle with unseen anomalies. The proposed NFM-DRA integrates DRA with a Normal Feature Memory, achieving new state-of-the-art performance and significantly improving robustness against unseen anomalies, highlighting the need for more robust evaluation metrics.
Retinal anomaly detection is crucial for identifying various eye and systemic diseases. However, progress in this field has been hampered by the absence of a comprehensive and publicly available benchmark for fair evaluation and advancement of new methods. Existing approaches often suffer from a limited range of anomaly types, test sets that are nearly saturated, and a lack of generalization evaluation, leading to less convincing experimental setups. Furthermore, many current benchmarks in medical anomaly detection primarily focus on one-class supervised methods, which train only with normal samples, overlooking the significant amount of labeled abnormal data and unlabeled data commonly found in clinical practice.
To address these critical gaps, researchers have introduced BenchReAD, a new benchmark for retinal anomaly detection. This benchmark is designed to be comprehensive and systematic in terms of both the data it includes and the algorithms it evaluates. BenchReAD incorporates two of the most widely used retinal imaging modalities: fundus photography and optical coherence tomography (OCT).
A Comprehensive Dataset for Robust Evaluation
From a data perspective, BenchReAD features significantly larger datasets with a more diverse array of anomaly categories compared to previous benchmarks. This ensures a more rigorous assessment of how well detection methods can generalize to new, unseen conditions. The test sets within BenchReAD include both ‘seen’ anomalies (those present in the training data) and ‘unseen’ anomalies (those absent from the training data), which is vital for evaluating real-world performance.
The fundus benchmark, for instance, is built from four public datasets, including a training set with over 23,000 normal and 16,000 abnormal images. Its test sets, RIADD and JSIEC, cover common retinal anomalies like diabetic retinopathy (DR), age-related macular degeneration (AMD), glaucoma, and myopia, alongside a wide range of unseen anomalies such as macular hole (MH) and branch retinal vein occlusion (BRVO).
Similarly, the OCT benchmark utilizes three public datasets, with a training set comprising over 25,000 normal and 53,000 abnormal images. Its test sets include OCT 2017, OCTDL, and OCTID, featuring both seen anomalies like choroidal neovascularization (CNV) and diabetic macular edema (DME), and unseen anomalies like epiretinal membrane (ERM) and central serous retinopathy (CSR).
Systematic Evaluation of Algorithms
From a methodological standpoint, BenchReAD systematically categorizes anomaly detection approaches into four groups based on their level of supervision: unsupervised, one-class supervised, semi-supervised, and fully supervised methods. This allows for a thorough comparison of different algorithmic paradigms.
Through extensive benchmarking, the researchers found that a fully supervised approach known as Disentangled Representations of Abnormalities (DRA) generally achieved the best performance. However, a significant limitation of DRA was its performance drop when encountering certain unseen anomalies. This is because DRA, being trained on both normal and abnormal samples, tends to rely on recognizing previously seen abnormal features rather than identifying deviations from normal features.
Introducing NFM-DRA: Enhancing Robustness for Unseen Anomalies
Inspired by memory bank mechanisms used in one-class supervised learning, the researchers proposed NFM-DRA (Normal Feature Memory-Disentangled Representations of Abnormalities). This novel approach integrates DRA with a Normal Feature Memory, which stores features of normal samples. By comparing test image features against this memory, NFM-DRA refines DRA’s predictions, mitigating the performance degradation when faced with unseen anomalies.
NFM-DRA has established a new state-of-the-art in retinal anomaly detection. It shows notable improvements, particularly on the RIADD and JSIEC fundus datasets, and performs comparably to DRA on OCT datasets, while significantly enhancing robustness for unseen anomalies like MH and ODE in RIADD, and certain blur fundus categories in JSIEC.
Also Read:
- Smart Search for Medical Images: How RadiomicsRetrieval Improves Diagnosis
- Advanced AI System Achieves High Precision in Wound and Scale Marker Segmentation
Rethinking Evaluation Protocols
The benchmark also highlights important insights into evaluation protocols. It confirms that the widely used OCT 2017 dataset is becoming saturated, meaning many methods achieve near-perfect scores on it, making it insufficient as a standalone benchmark for evaluating new methods. Furthermore, the paper emphasizes the importance of threshold-dependent metrics (like F1 scores, specificity, and sensitivity) in addition to threshold-independent ones (like AUC). While NFM-DRA achieves the highest average F1 scores, no single method consistently outperforms others across all datasets and anomaly categories, and threshold-dependent metrics show more significant fluctuations. This underscores the need for more attention to these metrics for real-world clinical deployment.
BenchReAD is publicly available, aiming to inspire rigorous evaluation and foster impactful research in retinal anomaly detection. For more details, you can refer to the full research paper: BenchReAD: A systematic benchmark for retinal anomaly detection.
