TLDR: This research introduces CLASP (Closed-Loop Autonomous Security Performance), a framework that aligns the security lifecycle with agentic capabilities to evaluate autonomous security agents. It also defines the Closed-Loop Capability (CLC) Score, a metric quantifying both loop closure and operational effectiveness. The paper surveys existing works, identifies key agentic drivers for success (like planning, reasoning, and error handling), and highlights critical gaps in current evaluation methods, proposing a blueprint for future closed-loop benchmarks to foster more robust and integrated security solutions.
The landscape of cybersecurity is undergoing a significant transformation, largely driven by the rapid adoption of AI-powered systems in offensive operations. As AI-driven attacks become more sophisticated and frequent, traditional defenses struggle to keep pace. Research and tools in cybersecurity often remain fragmented, focusing on isolated defensive functions, which creates vulnerabilities that adversaries can exploit. This challenge highlights a critical need for autonomous agents capable of integrating various security functions into a single, continuous ‘closed loop’ – from identifying threats to validating fixes.
However, the field has been missing three crucial elements: a clear framework to define the capabilities of these security agents across the entire security lifecycle, a standardized method for evaluating closed-loop agents, and a benchmark to measure their real-world performance. To address these gaps, researchers Mudita Khurana and Raunak Jain introduce CLASP—the Closed-Loop Autonomous Security Performance framework.
Introducing CLASP: A Unified Framework
CLASP provides a common language and rubric for assessing the capabilities of autonomous security agents. It aligns the key stages of the security lifecycle—reconnaissance, exploitation, root cause analysis, patch synthesis, and validation—with core agentic capabilities. These agentic capabilities include planning, tool use, memory, reasoning, reflection, and perception. By mapping these two dimensions, CLASP offers a detailed view of where current systems excel and where significant capability gaps still exist.
The framework was applied to 21 representative works in the field, revealing important insights into the strengths and weaknesses of existing autonomous security systems. This systematic survey helps researchers and practitioners understand which combinations of security functions and agentic capabilities lead to robust operations.
The Closed-Loop Capability (CLC) Score
Building on the CLASP framework, the authors define the Closed-Loop Capability (CLC) Score. This is a composite metric designed to quantify both how complete a security loop is (degree of loop closure) and how effective an agent is in its operations. The CLC Score aims to provide a balanced measure of an agent’s efficacy (its ability to achieve end-to-end, non-regressing completion efficiently) and its agentic efficiency (how appropriately and parsimoniously it uses its capabilities).
The CLC Score is calculated as a product of two components: an Efficacy Score, which measures the outcome, and an Agentic Efficiency Score, which evaluates the quality of the process. This multiplicative structure ensures that an agent doesn’t receive credit for an efficient failure, emphasizing the importance of both successful outcomes and intelligent resource utilization.
Key Insights from the Surveyed Agents
The research highlights several critical drivers for successful autonomous security agents:
- Planning & Reasoning: Agents with strong planning and reasoning capabilities consistently performed better across all security stages.
- Tool-Use with Planning: While tool-use is essential, it’s most effective when coupled with robust planning. Agents that planned their tool usage achieved significantly better results than those that simply used many tools without a coherent strategy.
- Reasoning for Analysis: Deep reasoning, such as chain-of-thought processes, showed the highest correlation with success in analytical stages like root cause analysis.
- Contextual Perception: Tasks requiring agents to perceive or discover information (e.g., scanning targets, reading logs) were more challenging than those where key information was provided upfront.
- Error Handling (Reflection & Adaptation): Agents with effective error handling and reflection mechanisms were better at recovering from unexpected tool outputs or failures, learning from mistakes, and adapting their strategies.
Also Read:
- TRACE: A Framework for Dynamically Evolving AI Agent Benchmarks
- SafeMind: A Framework for Secure Embodied AI Agents
Addressing Gaps and Future Directions
The paper identifies several gaps hindering the integration of security functions in current evaluations:
- Outcome-Only Scoring: Many evaluations focus solely on a single binary outcome, neglecting how the result was achieved or the operational risks involved.
- Episodic Resets: Benchmarks often treat each challenge as an isolated event, preventing agents from accumulating knowledge across tasks, which is crucial for human expertise.
- Disconnect from Enterprise Practice: Benchmarks frequently evaluate singular stages, ignoring the critical handoffs between functions that are common in real-world enterprise security pipelines.
To address these misalignments, the authors propose a blueprint for a closed-loop benchmark that emphasizes process quality through CLASP capability attribution, composed stages with persistent state, longitudinal memory, transparent validators, and realistic budgets and risk constraints.
In conclusion, CLASP and the CLC Score provide the necessary vocabulary, diagnostics, and measurements to advance both function-level performance and the overall effectiveness of closed-loop security agents. This work encourages the community to develop shared, capability-attributed benchmarks to move reliable closed-loop security agents from prototypes to deployable solutions.
For more in-depth information, you can read the full research paper here.
