TLDR: Microsoft has launched ExCyTIn-Bench, a new open-source benchmark designed to rigorously evaluate the performance of AI agents in conducting realistic cybersecurity investigations. This initiative aims to enhance the effectiveness and reliability of AI in threat detection and response by simulating complex, multi-stage cyberattacks within a controlled Azure environment, moving beyond traditional static knowledge assessments.
Microsoft has announced the release of ExCyTIn-Bench, an innovative open-source benchmark poised to revolutionize how artificial intelligence agents are evaluated for their capabilities in cybersecurity investigations. This new tool addresses a critical need in the evolving landscape of cyber defense, where AI’s role in detecting and responding to threats is becoming increasingly vital.
ExCyTIn-Bench distinguishes itself from conventional AI security benchmarks by focusing on dynamic, real-world scenarios rather than static threat intelligence trivia. According to Microsoft, the benchmark ‘aims to go beyond traditional AI security benchmarks that rely on threat intelligence trivia and other static knowledge by examining how agents take steps and use tools to examine data from realistic simulated attack scenarios.’ This approach allows for a more comprehensive assessment of an AI agent’s ability to reason, adapt, and utilize tools in the face of complex cyber incidents.
The benchmark’s robust methodology is built upon data derived from 57 log tables sourced from Microsoft Sentinel and related services. This extensive dataset was generated during eight simulated multi-stage attacks conducted on a controlled Azure tenant, meticulously designed to mimic a fictional company complete with users, groups, and applications. Researchers then leveraged this data to create bipartite alert-entity graphs, which in turn facilitated the generation of 589 question and answer pairs, along with detailed solution paths, to thoroughly test agents’ investigative prowess.
The ExCyTIn-Bench environment provides the question set alongside a MySQL database containing the simulated attack data, mirroring the resources available to a human analyst. AI agents under evaluation are tasked with querying this database to gather necessary information and are scored not only on the accuracy of their final answers but also on the logical steps taken to collect and synthesize relevant data.
Internally, Microsoft is already utilizing ExCyTIn-Bench to bolster its own AI-powered security features and in-house security-focused models, including the Microsoft Security Copilot. The company emphasizes that the benchmark is free and open-source, inviting AI developers and the broader cybersecurity community to perform their own benchmarks, contribute to its development, and share their findings.
Recent tests conducted using ExCyTIn-Bench have provided insightful performance metrics for various leading AI models. OpenAI’s GPT-5 in high reasoning mode demonstrated the strongest performance with an average reward score of 56.2%. It was followed by OpenAI-o3 at 45.6%, GPT-5 in low reasoning mode at 37.5%, GPT-5-mini at 36.9%, and GPT-o4-mini at 36.8%. Other models evaluated included xAI’s Grok 4 (34.4%), Alibaba’s Qwen 3-235b-thinking (30.2%), Meta’s Llama 4-17b-Maverick (29%), and Microsoft’s Phi-4-14B (8.5%). Notably, Google’s Gemini models were not included in these evaluations due to Google’s terms regarding benchmarking.
Also Read:
- OpenAI Addresses ‘ShadowLeak’ Zero-Click Flaw in ChatGPT’s Deep Research Agent Exposing Gmail Data
- AI Models Grok and DeepSeek Demonstrate Superiority in Cryptocurrency Market Predictions
For Chief Information Security Officers (CISOs) and IT leaders, ExCyTIn-Bench offers an objective and transparent mechanism to assess AI capabilities for security. It provides actionable insights into how AI tools reason through complex problems, aiding organizations in selecting solutions that genuinely enhance detection, response, and overall cyber resilience. Microsoft also plans to introduce personalized benchmarks in the near future, allowing for tailored evaluations specific to customer tenant threats.
