TLDR: Federated Learning (FL) enables collaborative AI model training without raw data sharing, enhancing privacy by design. However, it faces significant security threats like poisoning and backdoor attacks, and privacy threats such as data inference and membership inference. A recent survey comprehensively reviews over 200 papers on these attacks and their defense mechanisms, categorizing them and analyzing their trade-offs. It also explores FL frameworks and diverse real-world applications, emphasizing the critical need for integrated, scalable, and adaptive solutions to build robust and trustworthy FL systems, especially with the rise of Generative AI.
Federated Learning (FL) is a groundbreaking approach to artificial intelligence that allows multiple participants, or ‘clients,’ to collaboratively train a shared machine learning model without ever directly sharing their sensitive raw data. This design inherently boosts data privacy, as information remains on local devices or within organizations. However, despite this built-in privacy, FL systems are not immune to a wide array of security and privacy threats that can compromise their integrity and confidentiality.
A recent comprehensive survey, titled “On the Security and Privacy of Federated Learning: A Survey with Attacks, Defenses, Frameworks, Applications, and Future Directions,” delves into over 200 research papers to provide a detailed overview of the current state of attacks and defense mechanisms in FL. Authored by Daniel M. Jimenez-Gutierrez, Yelizaveta Falkouskaya, Jos´e L. Hernandez-Ramos, Aris Anagnostopoulos, Ioannis Chatzigiannakis, and Andrea Vitaletti, this work categorizes threats into security-enhancing and privacy-preserving techniques, offering a critical analysis of their strengths, limitations, and the trade-offs involved.
Understanding the Threats to Federated Learning
The distributed nature of FL introduces unique vulnerabilities. Security threats aim to undermine the model’s integrity or availability. These include:
- Poisoning Attacks: Adversaries inject malicious data or manipulate model updates to degrade overall performance, introduce biases, or insert hidden ‘backdoors.’ This can be done by altering local data (data poisoning) or directly manipulating model parameters (model poisoning).
- GAN-based Attacks: Malicious clients use Generative Adversarial Networks (GANs) to create realistic, manipulated data samples that can then be used for poisoning, making attacks harder to detect.
- Sybil Attacks: A single malicious entity creates multiple fake identities to gain disproportionate influence over the training process, amplifying their harmful contributions.
- Backdoor Attacks: A specific type of poisoning where an attacker secretly alters the model to behave maliciously only when triggered by a specific input, while performing normally otherwise.
- Free-Riding: Clients benefit from the global model without contributing meaningful updates, exploiting the collaborative nature of FL.
- Jamming Attacks: In wireless FL networks, adversaries can disrupt communication, preventing the exchange of critical model updates.
- Evasion Attacks: These occur during the model’s use, where carefully crafted inputs cause the model to make incorrect predictions without altering the training process itself.
- Straggling and Dropout: While not always intentional attacks, these refer to clients performing slowly or dropping out of the training process, which can be exploited by adversaries to disrupt convergence or introduce biases.
Privacy threats, on the other hand, focus on extracting sensitive information from the shared model updates or the final model itself:
- Gradient Inversion/Suppression: Attackers exploit shared gradients (the small adjustments models make during training) to reconstruct private training data or infer sensitive information.
- Reconstruction through Inference: Adversaries analyze model updates or outputs to reconstruct sensitive details about individual clients’ training data.
- Canary Gradient Attacks: An attacker injects subtle perturbations into gradients to deduce private data by observing the server’s response.
- Membership Inference: An adversary tries to determine if a specific data point was part of a client’s training dataset, potentially revealing sensitive personal information.
- Model Inconsistency: A malicious server distributes different model versions to clients, then analyzes the behavioral differences in their updates to infer private data.
- GANs-based Inference: Similar to GAN-based security attacks, GANs can be used to generate synthetic data that mimics the distribution of private client data, revealing sensitive patterns.
- Eavesdropping: An attacker intercepts communication between clients and the server to steal model information or infer sensitive data.
- Unintentional Data Leakage: This is a vulnerability where private training data inadvertently leaks through the gradient-sharing mechanism, which attackers can then exploit.
Defending Federated Learning
To counter these threats, various defense mechanisms have been developed:
- Robust Aggregation Operators: These algorithms modify how client updates are combined to minimize the impact of malicious or noisy contributions. Examples include Trimmed Mean, Median-based algorithms (like Geometric Median), and Krum, which select or filter updates based on their deviation from the norm.
- Asynchronous Schemes: To address issues like ‘stragglers’ (slow clients), these schemes allow the global model to update without waiting for all clients, improving efficiency and resilience.
- Pruning: This technique reduces model size and complexity by removing less important connections, which can also enhance security by eliminating neurons activated by malicious triggers.
- Adversarial Training: Clients generate adversarial examples locally to make their model updates more robust against attacks, improving the global model’s resilience.
- Cryptographic Approaches: These include:
- Zero-Knowledge Proofs (ZKPs): Allow clients to prove they’ve computed correct updates without revealing the underlying data.
- Oblivious Transfer (OT): Enables a client to obtain information from a server without revealing which information was chosen.
- Homomorphic Encryption (HE): Allows computations to be performed directly on encrypted data, keeping it private throughout the process.
- Secret Sharing (SS): Divides sensitive information into shares distributed among clients, so no single client can access the complete secret.
- Secure Multiparty Computation (MPC): Enables multiple parties to jointly compute a function on their private inputs without revealing those inputs to each other.
- Differential Privacy (DP): Adds carefully calibrated noise to model updates before they are shared, making it statistically difficult to infer information about individual data points.
- Gradient Clipping: Restricts the magnitude of updates from individual clients, preventing large, potentially malicious updates from disproportionately affecting the model.
- Personalized Solutions: Tailored defenses like FoolsGold (for Sybil attacks), FL-Defender (for poisoning), FLAME (for backdoors), and FLTrust (for malicious clients) offer targeted protection.
Frameworks and Applications
The survey also examines various FL frameworks like FEDML, Flower, and PySyft, comparing their support for privacy, security, attack simulation, and different FL types (horizontal, vertical, split). It highlights that while some frameworks offer comprehensive features, there’s still a need for more integrated attack-defense testbeds and support for advanced, adaptive defenses.
Federated Learning is being applied across numerous critical domains, including text prediction, healthcare (e.g., disease diagnosis without sharing patient records), finance (e.g., fraud detection across banks), intrusion detection systems, visual question-answering, autonomous vehicles, product manufacturing, mobile crowdsensing, merchandising, location services, and information retrieval. Each application area presents unique challenges and benefits from FL’s privacy-preserving capabilities.
Also Read:
- A New Defense Against Adversarial Attacks in Federated Learning
- Quantifying the Cost of Attack and Defense in Federated Learning Privacy
The Path Forward
The research paper concludes by outlining crucial future directions. A key takeaway is the need for integrated threat modeling that considers both privacy leakage and security compromises simultaneously, as attacks on one can amplify risks on the other. For instance, a poisoning attack could make a model more susceptible to data inference. Conversely, strong privacy defenses might inadvertently hinder the detection of malicious behavior.
Future research should focus on developing co-designed mechanisms that bridge this gap, such as privacy-aware robust aggregation and client-side collaborative monitoring. The integration of FL with Generative AI (GenAI) systems, including large language models, also presents new challenges, as these models are prone to data memorization, amplifying privacy leakage risks. Addressing these complex interactions is vital for building resilient and trustworthy federated systems in an increasingly data-driven world. For more in-depth technical details, you can refer to the full research paper available here.
