TLDR: A new research paper introduces two novel backdoor attacks, TrojanentRL and InfrectroRL, that exploit vulnerabilities in Deep Reinforcement Learning (DRL) systems beyond the traditional training phase. These attacks require significantly less access for adversaries and can persist through retraining or be injected into pre-trained models, with InfrectroRL even evading existing defenses. The findings highlight critical security gaps in the DRL supply chain, urging for more comprehensive defense strategies.
Deep Reinforcement Learning (DRL) systems are becoming increasingly vital in areas where safety is paramount, such as autonomous vehicles, nuclear fusion control, and cyber defense. However, new research highlights that their security remains largely unexplored, especially concerning hidden vulnerabilities known as backdoor attacks.
Traditionally, research on DRL backdoor attacks has focused on the training phase, assuming an adversary has full control over the training process. This often requires impractical levels of access, like infiltrating secure training pipelines or directly manipulating system memory, making these attacks less feasible in real-world scenarios.
A recent paper, titled “Beyond Training-time Poisoning: Component-level and Post-training Backdoors in Deep Reinforcement Learning”, challenges this narrow focus. Authored by Sanyam Vyas, Alberto Caron, Chris Hicks, Pete Burnap, and Vasilios Mavroudis from Cardiff University and The Alan Turing Institute, this work reveals critical vulnerabilities across the DRL supply chain, where backdoors can be embedded with significantly reduced adversarial privileges.
Introducing Novel Backdoor Attacks
The researchers introduce two innovative attacks: TrojanentRL and InfrectroRL. These attacks demonstrate how DRL systems can be compromised at different stages of their development and deployment, not just during initial training.
TrojanentRL exploits flaws at the component level. Instead of directly altering the policy network, this attack embeds a persistent backdoor within a fundamental DRL component, specifically the rollout buffer. This buffer is active throughout the training process. By replacing the standard buffer with a malicious version, the attack subtly manipulates reward signals based on predefined adversarial conditions and a trigger (like a white pixel in the corner of an input image). This manipulation steers the policy learning towards malicious actions. Crucially, any model trained, retrained, or fine-tuned with this compromised component will inherit the backdoor, making it highly persistent and difficult to detect through standard evaluations.
InfrectroRL represents a post-training backdoor attack, meaning it requires no access to the training, validation, or test data. This attack targets pre-trained models. Adversaries can intercept and tamper with models before deployment, for instance, by re-uploading malicious versions to public repositories under similar names. InfrectroRL injects backdoors by sparsely perturbing the model’s weights through targeted optimization. These perturbations remain dormant during normal operation but activate upon specific triggers, causing the agent to execute adversary-defined actions while maintaining plausible trajectories to evade detection. This method is computationally inexpensive, requiring GPU minutes instead of hours or days.
Why These Attacks Matter
The significance of TrojanentRL and InfrectroRL lies in their ability to operate under much stricter adversarial constraints compared to existing DRL backdoor attacks. They highlight that vulnerabilities exist not just in the training code, but also in the open-source components used to build DRL systems and in the packaging and distribution of pre-trained models.
Empirical Evaluation and Defense Evasion
The researchers rigorously evaluated both attacks across six Atari environments (Pong, Breakout, Qbert, Space Invaders, Seaquest, and Beam Rider) using standard backdoor metrics: Clean Data Accuracy (CDA), Attack Effectiveness Rate (AER), and Attack Success Rate (ASR).
The results show that TrojanentRL achieves performance comparable to or even superior to state-of-the-art training-time backdoor attacks like TrojDRL and BadRL, despite requiring significantly less adversarial access. InfrectroRL demonstrated near-perfect ASR and competitive AER across most environments, indicating high sensitivity to the trigger and significant degradation of agent behavior. It also maintained high model utility (CDA), signifying greater stealth.
Perhaps the most alarming finding is InfrectroRL’s ability to evade two leading DRL backdoor defenses, BIRD and SHINE. While these defenses successfully sanitized the baseline TrojDRL attack, InfrectroRL completely bypassed them across all tested Atari games. This underscores a substantial gap in current DRL backdoor detection approaches, which primarily focus on input observations for trigger identification.
Also Read:
- BackFed: A New Standard for Evaluating Backdoor Attacks in Federated Learning
- Unmasking the Flaw in LLM Prompt Injection Detection: A New Attack Evades State-of-the-Art Defenses
Conclusion and Future Outlook
This research exposes critical, previously underexplored vulnerabilities in the DRL supply chain. It demonstrates that backdoor attacks can be introduced beyond the training phase, persist through retraining, and even evade existing state-of-the-art defenses. These findings challenge prevailing security assumptions in DRL and emphasize the urgent need for more robust defenses. Future security measures must extend beyond just the training process to include supply-chain integrity verification, model provenance tracking, and runtime anomaly detection to effectively mitigate these stealthy and persistent threats.
