TLDR: A new research paper introduces ‘threat snapshots,’ a framework to identify and categorize LLM vulnerabilities in AI agents. This led to the creation of the ‘b3 benchmark,’ which used over 194,000 crowdsourced attacks to evaluate 31 popular LLMs. Key findings include that reasoning capabilities improve security, model size doesn’t correlate with it, and proprietary models generally offer better security, providing crucial insights for developing more secure AI agents.
As artificial intelligence agents, powered by large language models (LLMs), become increasingly common in our daily lives and industries, a critical question arises: how secure are they? A recent research paper, titled Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents, delves into this complex issue, offering a new framework and benchmark to systematically assess the security of the underlying LLMs that drive these agents.
The deployment of AI agents at scale has outpaced our understanding of how the choice of a backbone LLM impacts their security. These agents operate in a non-deterministic, sequential manner, making traditional security modeling difficult. Furthermore, the integration of AI components with conventional software means that novel LLM vulnerabilities can intertwine with existing security risks, creating a challenging landscape to navigate.
Introducing Threat Snapshots
To address these limitations, the researchers introduce a novel framework called ‘threat snapshots’. This approach focuses on isolating specific points in an agent’s execution where LLM vulnerabilities can manifest. By doing so, it allows for a systematic identification and categorization of security risks that originate from the LLM and propagate throughout the agent’s system. Unlike previous frameworks that either covered only specific vulnerabilities or required modeling entire agents, threat snapshots provide a clearer distinction between LLM-specific vulnerabilities and traditional software flaws, without needing to map out complete execution flows.
A threat snapshot captures the full context of a single LLM call, along with the attacker’s objective and method of attack. This includes details about the agent’s function, its state when attacked, the model context passed to the LLM, and a description of the threat itself, including how an attack is inserted and how its success is scored.
The b3 Benchmark: A New Standard for Agent Security
Using the threat snapshot framework, the team constructed the ‘b3 benchmark’ (backbone breaker benchmark). This benchmark is built upon 10 distinct threat snapshots, each designed with three levels of defense (L1, L2, L3) to represent varying security constraints. To ensure the benchmark was robust, it utilized an impressive dataset of 194,331 unique crowdsourced adversarial attacks, gathered through a gamified red-teaming challenge. This method allowed for the collection of high-quality, context-dependent attacks that static datasets often miss.
The b3 benchmark was then used to evaluate 31 popular LLMs, providing crucial insights into their security strengths and weaknesses when acting as agent backbones.
Key Findings from the Evaluation
The evaluation revealed several significant findings:
- Reasoning Improves Security: One of the most striking observations was that enabling reasoning capabilities in LLMs generally led to improved security. This suggests that models capable of more sophisticated thought processes are better at resisting adversarial attacks.
- Model Size Doesn’t Correlate with Security: Surprisingly, unlike performance benchmarks where larger models often excel, the research found no significant correlation between an LLM’s size and its security. Larger versions of models without reasoning capabilities showed no clear advantage, and sometimes even performed worse than their smaller counterparts.
- Closed-Weight Systems Lead: Models with closed weights (proprietary systems) generally outperformed open-weight models in terms of security. However, it’s important to note that closed-weight systems often include additional guardrails and safety layers beyond the base model, which contributes to their enhanced security.
- Consistency Across Defenses: The most secure models maintained their top positions across different defense levels (weak, strong, and self-judging), indicating that the type of defense employed doesn’t drastically alter the relative security of the LLM backbone.
- Security vs. Safety: The benchmark highlighted that LLM security is distinct from broader safety concerns (like toxicity). Different models excelled in tasks involving content safety abuse compared to overall security.
- Task-Specific Security: A model’s security properties can vary significantly depending on the task type (e.g., direct instruction override, indirect tool invocation). This suggests that LLM backbones should be chosen with specific use cases in mind.
Also Read:
- Unmasking a Hidden Threat: How Prompt Compression Exposes LLM Agents to New Attacks
- Prompt Quality’s Hidden Impact on AI-Generated Code Security
Implications for AI Agent Development
This research provides a foundational tool for treating security as a primary dimension in LLM evaluation, alongside existing capability benchmarks. The b3 benchmark offers actionable insights for developers to select the most secure backbone LLM for their specific AI agent applications. While the study focused on backbone LLMs, the threat snapshot framework can also guide future work in red-teaming and evaluating external defenses in agentic systems, paving the way for more secure and robust AI agents.
