TLDR: CTIARENA is a new benchmark for evaluating Large Language Models (LLMs) on complex cyber threat intelligence (CTI) tasks. It addresses limitations of previous benchmarks by using heterogeneous, multi-source CTI and knowledge-augmented settings. The research found that LLMs perform poorly in closed-book scenarios but significantly improve with domain-specific knowledge augmentation, highlighting the need for tailored techniques in cybersecurity AI.
In the rapidly evolving world of cybersecurity, understanding and responding to threats is paramount. This is where Cyber Threat Intelligence, or CTI, plays a crucial role, offering vital insights to detect and neutralize emerging dangers. With the impressive language understanding and reasoning abilities of Large Language Models (LLMs), there’s a growing interest in using them for CTI. However, to truly harness their potential, we need robust ways to measure how well they perform.
Previous attempts to evaluate LLMs in CTI have faced several limitations. Many relied solely on the models’ pre-trained knowledge (a “closed-book” setting), without tapping into the vast, specialized CTI knowledge bases available. They also covered only a small range of tasks and often restricted evaluations to single sources of information. Real-world cybersecurity, however, demands reasoning across many different and often complex sources.
Introducing CTIARENA: A New Standard for LLM Evaluation in CTI
To bridge these gaps, researchers have introduced CTIARENA, a groundbreaking benchmark designed to rigorously evaluate LLM performance on diverse, multi-source cyber threat intelligence. Unlike its predecessors, CTIARENA operates in “knowledge-augmented” settings, meaning LLMs are given access to relevant CTI knowledge bases, mimicking how human analysts work.
CTIARENA organizes the CTI analysis landscape into nine distinct tasks, categorized into three main types:
- Structured Tasks: These involve reasoning over authoritative taxonomies like CVE (Common Vulnerabilities and Exposures), CWE (Common Weakness Enumeration), CAPEC (Common Attack Pattern Enumeration and Classification), and MITRE ATT&CK. These frameworks catalog vulnerabilities, weaknesses, and adversary behaviors in standardized formats. Examples include mapping a vulnerability to its root cause or deriving attack techniques.
- Unstructured Tasks: These focus on interpreting narrative threat reports and blogs from security vendors. Analysts must extract and synthesize insights about adversaries, campaigns, and malware families from free-text descriptions. Tasks here include constructing campaign storylines, profiling threat actors, and analyzing malware evolution.
- Hybrid Tasks: These combine both structured and unstructured intelligence, bridging the gap between narrative reports and formal taxonomies. For instance, attributing attack behaviors described in a blog to specific MITRE ATT&CK techniques or correlating vulnerability descriptions in reports with CWE entries.
Key Findings: The Power of Knowledge Augmentation
The evaluation of ten widely used LLMs within CTIARENA revealed some critical insights. Most LLMs struggled significantly in closed-book setups, often hallucinating or providing incorrect information. This highlights that general-purpose LLMs, while powerful, lack the specialized, long-tail knowledge crucial for CTI without external help.
However, when augmented with security-specific knowledge through specially designed retrieval techniques, these models showed remarkable improvements. Simple knowledge injection was often enough for structured tasks, bringing accuracy close to perfect. For more complex hybrid and unstructured tasks, generic retrieval methods (like standard RAG) sometimes fell short. This is where CTIARENA’s tailored techniques, such as CSKG-guided RAG (leveraging a Cyber Security Knowledge Graph) and query-expanded RAG (decomposing narrative inputs into fine-grained behaviors), proved particularly effective, significantly boosting performance by better aligning free text with formal CTI taxonomies.
The research also delved into common failure modes, such as “semantic drift” where models were misled by textually similar but conceptually misaligned retrieved information, or instances where models retrieved correct evidence but failed to utilize it. These observations underscore the need for more robust and domain-aware augmentation strategies.
Also Read:
- Evaluating AI’s Cyber Exploitation Capabilities with PACEbench
- Evaluating AI Agents on Complex Scientific Research Tasks
The Future of AI in Cybersecurity
The findings from CTIARENA emphasize that simply scaling up LLM size isn’t enough for systematic progress in CTI. Instead, the future lies in knowledge-augmented LLMs equipped with retrieval strategies specifically tailored to the unique structure and semantics of cyber threat intelligence. CTIARENA is set to be a vital tool for future research, driving the development of next-generation CTI copilots that can truly assist human analysts in navigating the complex threat landscape. You can read the full research paper here: CTIARENA: Benchmarking LLM Knowledge and Reasoning Across Heterogeneous Cyber Threat Intelligence.
