TLDR: FedOnco-Bench is a new, reproducible benchmark designed to evaluate federated learning (FL) methods for tumor segmentation using synthetic CT data. It assesses both segmentation accuracy and privacy leakage through membership inference attacks. The study reveals a clear trade-off: high-performing FL methods like FedAvg show significant privacy risks, while Differential Privacy (DP-SGD) drastically reduces privacy leakage at the expense of some accuracy. FedProx offers a balanced compromise. This open-source benchmark provides a standardized platform for developing and comparing privacy-preserving FL algorithms in medical imaging.
Federated Learning (FL) is an innovative approach that allows multiple institutions, such as hospitals, to collaboratively train machine learning models without directly sharing sensitive patient data. This method is particularly valuable in privacy-sensitive fields like healthcare, where data privacy is paramount. While FL significantly reduces privacy risks compared to traditional centralized training, it’s not entirely immune to threats like membership inference attacks, where an attacker might deduce if a specific patient’s data was part of the training set. Additionally, FL faces challenges with data heterogeneity, meaning data can vary significantly across different institutions due to differences in equipment or patient demographics.
To address these challenges and provide a standardized platform for evaluating privacy-aware FL, researchers have introduced FedOnco-Bench. This reproducible benchmark focuses on privacy-aware federated tumor segmentation using synthetic oncologic CT scans with tumor annotations. The use of synthetic data is crucial as it allows for public sharing and evaluation without compromising actual patient privacy.
The FedOnco-Bench evaluates several common FL methods: FedAvg, FedProx, FedBN, and FedAvg enhanced with Differential Privacy (DP-SGD). FedAvg is a foundational FL algorithm that averages model updates from clients. FedProx introduces a regularization term to help stabilize training with diverse data, while FedBN is designed to handle feature differences across institutions by keeping batch normalization statistics local. DP-SGD is a robust privacy-preserving technique that adds carefully calibrated noise to gradients during training to limit privacy leakage.
The benchmark assesses two critical aspects: segmentation performance, measured by the Dice coefficient and cross-entropy loss, and privacy leakage, quantified by the Area Under the Curve (AUC) of a membership inference attack. A Dice coefficient closer to 1.0 indicates better segmentation accuracy, while an MIA AUC closer to 0.5 suggests better privacy (random guessing), and closer to 1.0 indicates significant leakage.
Key Findings and the Privacy-Utility Tradeoff
The results from FedOnco-Bench highlight a distinct trade-off between privacy and utility (model performance). Methods like FedAvg and FedBN achieved high segmentation performance, with Dice scores around 0.85, comparable to a centralized model trained on all pooled data. However, these methods also showed significant privacy leakage, with MIA AUCs around 0.72. This indicates that even without sharing raw data, these models can inadvertently memorize enough information to be vulnerable to privacy attacks.
In contrast, FedAvg with DP-SGD provided a much higher level of privacy, achieving an MIA AUC of approximately 0.25, meaning an attacker’s ability to infer membership was close to random chance. This enhanced privacy, however, came at a cost to accuracy, with the Dice score dropping to about 0.79. This clearly illustrates the classic privacy-utility tradeoff in machine learning.
FedProx offered a balanced performance, slightly reducing privacy risk (MIA AUC around 0.68) compared to FedAvg and FedBN, while maintaining a respectable Dice score of 0.84. This suggests that its regularization helps mitigate overfitting and, consequently, privacy leakage, without a drastic drop in accuracy.
Also Read:
- Advancing Medical Image Analysis with Adaptable Foundation Models
- LSHFed: Securing Federated Learning with Efficient Gradient Verification
Implications and Future Directions
FedOnco-Bench serves as a crucial open-source platform for benchmarking and developing privacy-preserving FL methods for medical image segmentation. It provides a standardized framework for researchers to compare new algorithms against established baselines, considering both performance and privacy. The findings underscore that the choice of FL algorithm depends on the specific privacy requirements of a medical application. If privacy is paramount, a method like DP-SGD might be necessary, even with a reduction in accuracy. If some privacy risk is acceptable for higher accuracy, non-private FL methods could be considered.
The benchmark’s use of synthetic data ensures full reproducibility and public accessibility, overcoming the significant hurdles of sharing real patient data. Future work can expand on this by exploring more sophisticated differential privacy methods, investigating other privacy threats beyond membership inference, and incorporating more realistic scenarios like partial client participation and communication constraints. The full implementation of FedOnco-Bench is open-source and can be downloaded from GitHub.
