TLDR: A survey of over 150 papers details the emerging field of AI agent security, categorizing it into applications (how agents are used in cybersecurity), threats (vulnerabilities they possess), and defenses (countermeasures). It highlights trends like the shift to multi-agent architectures and the dominance of GPT models, while also identifying critical research gaps in model and modality coverage, and benchmark fragmentation.
The world of cybersecurity is undergoing a significant transformation with the rise of autonomous Large Language Model (LLM) agents, often simply called AI agents. These intelligent systems are moving beyond just understanding and generating text; they can now plan, execute actions, use tools, and adapt to external environments. This newfound capability, while powerful for both offensive and defensive security operations, also introduces a new set of complex security risks.
A recent comprehensive survey, “A Survey on Agentic Security: Applications, Threats and Defenses,” delves into this evolving landscape, providing a holistic view of agentic security. It structures the field around three core pillars: how agents are applied in security, the threats they face, and the defenses designed to protect them. This work analyzes over 150 papers, offering a detailed taxonomy and cross-cutting analysis of emerging trends and critical research gaps. You can read the full paper here: A Survey on Agentic Security.
Applications: How AI Agents are Used in Security
AI agents are being deployed across the entire cybersecurity lifecycle, demonstrating remarkable capabilities. In offensive security, often called ‘red-teaming,’ agents are used for autonomous penetration testing, vulnerability discovery, fuzzing, and exploit generation. They can simulate adversarial workflows, identify bug-prone regions, and even generate environment-aware exploits or malware. Examples include systems that perform end-to-end penetration tests or guide fuzzing to uncover flaws.
For defensive security, or ‘blue-teaming,’ agents are crucial for continuous monitoring, threat detection, incident response, threat hunting, and automated patching. They can act as intelligent Security Operations Center (SOC) frameworks, analyzing alerts, triaging threats, and executing response playbooks. Agents are also used for intelligent threat hunting, analyzing unstructured intelligence to uncover hidden adversaries, and for automated forensics to reconstruct attacks and generate reports. Furthermore, they can autonomously detect and remediate vulnerabilities through automated patch synthesis and validation.
Beyond general security, agents are also tailored for domain-specific applications. This includes cloud and infrastructure security (e.g., automated scanning, hardening, and remediation), web and application security (e.g., multi-agent web penetration testing, secure automation in web environments), and specialized areas like finance (smart contract auditing) and healthcare (HIPAA-compliant agent frameworks).
Threats: The Vulnerabilities of Agentic Systems
The transition to autonomous agents significantly increases their vulnerability compared to passive LLMs. The survey identifies several critical attack surfaces:
- Injection Attacks: These involve embedding malicious instructions within the prompt to manipulate the agent into unintended actions. This includes direct prompt injection, indirect prompt injection (where malicious content is hidden in data the agent processes), and memory injection attacks that poison an agent’s long-term memory.
- Poisoning and Extraction Attacks: Poisoning attacks corrupt an agent’s memory or knowledge retrieval systems, causing it to execute malicious actions. Model extraction attacks involve repeatedly querying an agent’s API to train an unauthorized ‘clone,’ effectively stealing intellectual property.
- Jailbreak Attacks: These bypass a model’s built-in safety measures to force it to produce harmful content. AI agents are often more vulnerable to jailbreaks than their underlying LLMs, with simple chatbot jailbreaking techniques proving effective against browser agents.
- Agent Manipulation Attacks: These target the agent’s higher-level cognitive functions like planning and reasoning. Goal hijacking subtly alters an agent’s objectives, while action hijacking tricks an agent into assembling seemingly benign data into harmful instructions. Reward hacking exploits reward mechanisms in agents, causing them to maximize scores without true alignment.
- Red-Teaming Attacks: Adversarial agents can be used to intercept and manipulate inter-agent communications, simulating multi-turn interactions to discover emergent risks.
Defenses: Hardening AI Agent Systems
To counter these threats, a growing body of research focuses on developing robust defense mechanisms:
- Secure-by-Design: This involves embedding layered verification, isolation, and control-flow integrity into agent architectures from the outset. Modular and plan-execute isolation can significantly reduce injection rates. Governance-oriented frameworks extend these principles through trust calibration and layered threat modeling.
- Multi-Agent Security: For systems with multiple agents, zero-trust principles and dynamic collaboration are applied to minimize information leakage under adversarial conditions. Formal cross-agent verification and debate-based collectives are used to improve detection and consensus.
- Runtime Protection: This includes reasoning- and knowledge-enhanced guardrails that reduce jailbreak failures. Adaptive systems can sustain accuracy under evolving threats, while human-in-the-loop oversight embeds runtime policy enforcement and approval gates for accountability. Behavioral anomaly detectors monitor agent actions for suspicious patterns.
- Security Operations: Formal verification systems ensure behavioral correctness, while LLM-driven analyzers assist in static analysis for vulnerability detection. Autonomous response pipelines fuse LLM reasoning with threat intelligence to reduce incident response times. Cloud-native forensic systems automate evidence extraction and improve causal reconstruction.
Also Read:
- Unmasking the Fragility of AI Agents: A New Framework for Tool Selection Security
- AI Agents Vulnerable to Stealthy Backdoors Through Data Poisoning
Cross-Cutting Analysis and Emerging Trends
The survey’s analysis of over 150 papers reveals several key trends. Architecturally, there’s a clear shift from monolithic designs (single LLM for end-to-end reasoning) to planner-executor architectures and hybrid models, which allow for modularity and improved interpretability. Executor and planner roles dominate, reflecting the operational emphasis on task decomposition and control, with a rising presence of critics/verifiers.
In terms of LLM backbones, GPT-family models are prevalent, raising concerns about monoculture and reproducibility, though open-weight backbones like Claude and LLaMA are gaining traction. Text remains the primary input modality, but there’s increasing diversification into structured data like logs and code. Finally, static pre-trained knowledge is still the dominant knowledge source, with limited adoption of adaptive learning paradigms like Retrieval-Augmented Generation (RAG) or fine-tuning, which could impact resilience in dynamic threat environments.
This survey underscores the critical need for continued research in agentic security, particularly in addressing cross-domain challenges, the economics of agentic security, and prioritizing defense techniques with provable safety guarantees to build trust and resilience in these powerful new systems.
