TLDR: FeDABoost is a new Federated Learning framework that improves model performance and fairness in diverse data environments. It uses an adaptive aggregation method to give more weight to high-performing clients and a dynamic boosting mechanism (focal loss) to help underperforming clients improve. Tested on MNIST, FEMNIST, and CIFAR10, FeDABoost consistently showed better fairness and competitive accuracy compared to existing methods like FedAvg and Ditto.
Federated Learning (FL) has emerged as a powerful approach to training machine learning models collaboratively across many devices, like smartphones or medical sensors, without centralizing sensitive user data. This decentralized method helps address significant privacy, security, and computational challenges associated with traditional centralized learning. Instead of sending raw data to a central server, only model updates are shared, keeping personal data localized on devices.
However, FL faces critical hurdles, especially when dealing with non-Independent and Identically Distributed (non-IID) data. This means that the data on each client device can vary significantly in terms of distribution, class imbalances, or unique challenging examples. Such variations can lead to inconsistent local model updates, hindering the global model’s ability to generalize effectively. Another pressing concern is ensuring fairness in client participation. Current FL methods often inadvertently favor clients with more resources or higher data quality, potentially marginalizing others and leading to biased global models.
To tackle these challenges, researchers have introduced FeDABoost: Fairness Aware Federated Learning with Adaptive Boosting, a novel FL framework designed to enhance both performance and fairness in non-IID settings. This innovative approach integrates a dynamic boosting mechanism and an adaptive gradient aggregation strategy, drawing inspiration from the Multiclass AdaBoost (SAMME) algorithm.
How FeDABoost Works
FeDABoost operates on two main fronts to achieve its goals. First, it employs an adaptive aggregation method that assigns higher weights to clients with lower local error rates. This means that clients whose local models perform well contribute more significantly to the global model, promoting more reliable and accurate updates. This mechanism helps to down-weight less reliable contributions, especially in diverse data environments.
Second, FeDABoost dynamically boosts underperforming clients. It achieves this by adjusting the focal loss focusing parameter during local training. Focal loss is a technique that emphasizes “hard-to-classify” examples, ensuring that clients struggling with particular data points receive targeted attention. By increasing this parameter for underperforming clients, FeDABoost encourages them to improve their models on challenging data, thereby reducing performance disparities across the network.
These two mechanisms work in tandem. While strong clients contribute more to the global model, weaker clients are actively supported to improve, leading to a more balanced and fair overall system. The framework aims to reduce the variance in model performance across clients while maintaining high average predictive accuracy.
Also Read:
- AutoMaAS: A Self-Evolving Framework for Multi-Agent AI Systems
- BayesianRouter: A Smart Approach to Aligning Language Models with Human Preferences
Experimental Validation
The effectiveness of FeDABoost was rigorously evaluated on three benchmark datasets: MNIST, FEMNIST, and CIFAR10. These datasets represent various levels of data heterogeneity, with MNIST and CIFAR10 simulating non-IID conditions and FEMNIST being inherently non-IID. The performance of FeDABoost was compared against two widely used FL baselines: FedAvg and Ditto.
The experimental results demonstrated that FeDABoost consistently achieves improved fairness and competitive performance. For instance, on the MNIST dataset, FeDABoost showed higher F1 scores and reached these scores in fewer communication rounds compared to FedAvg, indicating better communication efficiency. On FEMNIST, FeDABoost, especially when configured with the AdamW optimizer, significantly outperformed both FedAvg and Ditto, achieving faster and more stable convergence. The study also highlighted that FeDABoost resulted in a more concentrated distribution of F1 scores across clients, signifying greater consistency and fairness.
Quantifying fairness, FeDABoost achieved a significant reduction in the variance of F1 scores across clients on both MNIST and FEMNIST datasets compared to the baselines. This confirms that the framework not only enhances overall performance but also ensures a more equitable performance distribution among participating clients.
The research paper, available at arXiv, concludes that FeDABoost is particularly well-suited for cross-silo FL settings where fairness and the interpretability of client contributions are paramount. While acknowledging some sensitivity to hyperparameters, the authors plan to explore further enhancements in future work, including alternative boosting strategies and adaptive mechanisms for greater stability and generalization.
