TLDR: A new study introduces Single-Task Poisoning (STP), a realistic data poisoning threat to continual learning (CL) where an attacker only has access to a single task’s data. The research demonstrates that even under these limited conditions, such attacks can severely degrade a CL model’s performance on both previously learned and future tasks. It also proposes a defense framework using task vectors to detect and mitigate these attacks.
Continual learning (CL) is a fascinating area of artificial intelligence where models learn continuously from a stream of data, adapting to new information without forgetting what they’ve already learned. This process is crucial for AI systems that need to operate in dynamic environments, constantly updating their knowledge. However, like many advanced technologies, CL is not immune to security threats, particularly data poisoning.
Data poisoning involves intentionally manipulating training data to negatively impact a machine learning model’s predictions. While previous research has explored various data poisoning scenarios in CL, many of these assumed that attackers had extensive knowledge or access to the model and multiple data tasks. This new research, detailed in the paper “Addressing the Devastating Effects of Single-Task Data Poisoning in Exemplar-Free Continual Learning”, introduces a more realistic and constrained threat model: Single-Task Poisoning (STP).
Understanding Single-Task Poisoning (STP)
In the STP setup, an adversary has very limited access. They can only manipulate data for the current task in the data stream, without any knowledge or access to the model itself, or to any previous or future tasks. This makes the attack scenario much more plausible in real-world applications where data might come from various, potentially untrusted, sources.
The study demonstrates that even under these stringent conditions, adversaries can significantly compromise a model’s performance using standard image corruptions. The goal of STP attacks isn’t just to disrupt the current task, but to severely impact the entire continual training process. This includes decreasing the model’s ‘stability’ (its performance on past tasks, leading to catastrophic forgetting) and its ‘plasticity’ (its capacity to adapt to and learn new tasks).
How STP Attacks Work and Their Impact
The researchers proposed two main types of STP attacks: ‘uniform’ (BASE) and ‘classifier bait’ (BAIT). BASE attacks contaminate all classes in a task with a single type of image corruption, like Gaussian blur. BAIT attacks are more subtle, corrupting only a subset of classes within a task to create misleading data distribution differences, essentially ‘baiting’ the classifier to focus on these artificial correlations rather than true class features.
The findings are quite stark. Poisoning a single task in a CL sequence has far more severe consequences than in traditional joint training. In joint training, the model is exposed to both clean and poisoned data simultaneously, often mitigating the impact on unpoisoned classes. However, in CL, training on an isolated poisoned task without access to clean samples from previously seen classes can significantly degrade performance on both past and future tasks. While the poisoned task itself suffers the most, the ripple effect on stability and plasticity is profound.
Interestingly, BAIT attacks, despite poisoning fewer data samples, were found to be more effective than BASE attacks in affecting the model’s performance on future tasks. This suggests that creating spurious correlations can be more damaging to future learning than a general data distribution shift. The study also found that using multiple types of corruptions (MULTIBASE and MULTIBAIT attacks) was generally less effective than single-corruption attacks, and that a significant percentage of the task’s data must be poisoned for the attack to be truly effective.
Also Read:
- New Research Uncovers Stealthy Backdoors in Deep Reinforcement Learning Supply Chains
- BackFed: A New Standard for Evaluating Backdoor Attacks in Federated Learning
Developing Defenses Against STP
A key challenge for defending against STP attacks is their stealthiness. The poisoned tasks can still achieve reasonable validation accuracy, making them hard to detect by simply monitoring current task performance. This is because the validation set itself might also be poisoned.
To counter this, the researchers propose a high-level defense framework involving three steps: checkpointing the model’s weights before training on a new task, conducting a poison detection test after training, and optionally rolling back the model’s weights if poisoning is detected. The core of this defense is a novel poison detection method based on ‘task vectors,’ which capture the changes in a neural network’s parameters after training on a specific task. By comparing these vectors, the system can identify anomalies indicative of poisoning.
The defense evaluation showed that this task-vector-based method can correctly identify most poisoned tasks, though BAIT attacks proved harder to detect due to their more subtle data manipulation. This research highlights the critical need for robust defense mechanisms in continual learning to ensure the security and reliability of AI systems that learn and adapt over time.
