TLDR: AdvEvo-MARL is a novel multi-agent reinforcement learning framework that enhances the safety and robustness of LLM-based multi-agent systems. It achieves this by co-evolving ‘attacker’ agents that generate sophisticated threats and ‘defender’ agents that learn to resist these attacks while maintaining task performance. This approach internalizes safety, significantly reduces attack success rates, and preserves or improves task utility without relying on external guard modules or incurring additional system overhead.
Large Language Model (LLM)-based multi-agent systems are becoming increasingly sophisticated, excelling at complex tasks like planning, using various tools, and coordinating roles. However, their open nature and intricate interactions also expose them to significant security risks, including jailbreak attacks, prompt injections, and adversarial collaboration, where malicious actors try to manipulate the system.
Current defense strategies typically fall into two categories: either individual agents try to filter out unsafe instructions themselves (self-verification), or external modules act as guards to police behaviors. Both approaches have notable drawbacks. Self-verification often fails because a single agent lacks the broader context to detect complex, cross-agent attack chains or risks that arise from delegation. External guard modules, while effective to some extent, add system overhead and create a single point of failure – if the guard is compromised, the entire system’s safety collapses. Scaling up these guards also quickly becomes prohibitively expensive and complex.
To address these critical challenges, researchers have introduced AdvEvo-MARL, a novel co-evolutionary multi-agent reinforcement learning framework. This innovative approach aims to embed safety awareness directly into the task-performing agents themselves, rather than relying on external security layers.
How AdvEvo-MARL Works
AdvEvo-MARL operates by jointly optimizing two types of agents in an adversarial learning environment: attackers and defenders. The attackers are trained to synthesize increasingly sophisticated jailbreak prompts, constantly evolving their methods to bypass safety measures. Simultaneously, the defenders, which are the task agents, are trained to both successfully accomplish their assigned duties and robustly resist these evolving attacks.
This framework unfolds in two main stages. First, an “attacker warm-up” phase uses supervised fine-tuning to give the attacker agents initial knowledge of jailbreak strategies, preventing them from generating trivial or ineffective attacks at the start. Following this, an “adversarial co-evolutionary RL stage” begins, where attackers and defenders are continuously optimized through repeated interactions. This dynamic process enables defenders to develop highly robust and adaptive safety behaviors against a constantly changing threat landscape.
A key innovation within AdvEvo-MARL is the introduction of a “public baseline” for advantage estimation. Agents within the same functional group (e.g., all attackers or all defenders) share a group-level mean-return baseline. This mechanism helps stabilize the learning process, reduces variance in policy updates, and fosters stronger cooperation among agents within their respective groups.
Reward System and Training
To align with the distinct objectives of attackers and defenders, AdvEvo-MARL employs separate reward mechanisms. Attackers receive rewards based on whether the system’s final output achieves their malicious goal. Defenders, on the other hand, are rewarded for both resisting jailbreak attempts and successfully completing their tasks. This dual reward structure ensures that defenders are incentivized to maintain both safety and utility.
The training process involves multiple distinct backbone models collaborating under reinforcement learning, ensuring genuine co-evolution across diverse agents, rather than relying on a single shared-parameter model.
Experimental Results and Impact
Extensive experiments were conducted across three representative multi-agent attack scenarios: agent manipulation, message corruption, and user instruction hijacking. These were tested on various system topologies, including chain, tree, and complete graph modes, using different LLM backbones.
The results demonstrate that AdvEvo-MARL consistently achieves significantly lower attack success rates (ASR) and contagion rates (CR) compared to existing baselines. In many configurations, it kept the ASR below 10% and even matched or outperformed proprietary models like GPT-4o-mini in simpler topologies. Crucially, AdvEvo-MARL also preserved, and in some cases even improved, task accuracy (up to +3.67% on reasoning tasks). This indicates that safety and utility can be jointly enhanced without the need for additional guard agents or increased system overhead.
The research highlights that training with dynamic, evolving attackers leads to more robust defenders. Furthermore, training defenders in a multi-agent setting yields superior benefits over training them individually, fostering collaborative defense behaviors that are difficult to achieve in isolation. The public baseline mechanism was also shown to lead to more stable and efficient learning dynamics, improving both task utility and system safety.
Also Read:
- SafeEvalAgent: A Dynamic Approach to AI Safety Evaluation
- Unforeseen Dangers: How Self-Evolving AI Agents Can Develop Harmful Behaviors
Conclusion
AdvEvo-MARL represents a significant step forward in building inherently safe and capable multi-agent systems. By co-training attackers and defenders in a dynamic adversarial environment, the framework enables agents to continuously adapt to evolving threats, developing stronger and more generalizable defense capabilities. This approach offers a promising and unified framework for developing robust and performant AI agent teams. You can read the full research paper here.
