Benchmarking LLMs for Cyber Threat Intelligence: Introducing AthenaBench

TLDR: AthenaBench is a new dynamic benchmark for evaluating Large Language Models (LLMs) in Cyber Threat Intelligence (CTI). It improves upon previous benchmarks with a new task (risk mitigation), dynamic data from live CTI sources, and enhanced evaluation metrics. The study found that proprietary LLMs like GPT-5 perform best, but all models struggle with complex reasoning tasks like threat actor attribution and risk mitigation, highlighting a need for more specialized CTI models. Web search capabilities significantly improve performance.

Large Language Models, or LLMs, have shown impressive abilities in understanding and reasoning with natural language. However, their full potential in Cyber Threat Intelligence (CTI) has remained largely untapped. CTI is a critical field where analysts sift through vast amounts of unstructured reports to extract actionable insights, a process that LLMs could significantly streamline.

A new research paper introduces AthenaBench, an advanced benchmark designed to rigorously evaluate LLMs in the complex domain of CTI. This work builds upon previous efforts like CTIBench, aiming to provide a more comprehensive and dynamic assessment of LLM capabilities.

What Makes AthenaBench Unique?

AthenaBench distinguishes itself through several key enhancements:

• Improved Task and Dataset Design: It introduces a new task focused on Risk Mitigation Strategies (RMS), which evaluates an LLM’s ability to suggest effective defensive measures. The benchmark also refines existing tasks by removing duplicate data, reducing contamination risks, and enhancing areas like threat actor attribution and attack technique extraction. It uses a larger, more current set of examples for each task to better reflect real-world CTI practices.
• Dynamic Data Construction: Unlike benchmarks that rely on static information, AthenaBench continuously generates benchmark samples by leveraging live CTI data sources and APIs, such as MITRE ATT&CK and the NVD API. This ensures that the evaluations remain relevant to emerging vulnerabilities, techniques, and threat actors, keeping pace with the rapidly evolving cyber threat landscape.
• Enhanced Evaluation Metrics: The benchmark features improved task-specific evaluation metrics and introduces a unified aggregated score. This provides a holistic view of an LLM’s CTI reasoning capabilities, highlighting their strengths and weaknesses across different tasks.

The Six Core Tasks of AthenaBench

AthenaBench evaluates LLMs across six distinct CTI reasoning tasks:

• CTI Knowledge Test (CKT): Assesses an LLM’s foundational knowledge of CTI frameworks, concepts, and best practices.
• Root Cause Mapping (RCM): Challenges LLMs to identify the underlying cause of a vulnerability by mapping its description to the correct Common Weakness Enumeration (CWE) category.
• Vulnerability Severity Prediction (VSP): Evaluates an LLM’s ability to determine the severity of a vulnerability from its description, predicting its CVSS v3.1 base vector.
• Threat Actor Attribution (TAA): Requires LLMs to infer which adversary is responsible for observed activity patterns, without explicit name mentions.
• Risk Mitigation Strategy (RMS): Tests an LLM’s capacity to recommend appropriate MITRE ATT&CK mitigation strategies based on a detailed attack scenario.
• Attack Technique Extraction (ATE): Focuses on whether LLMs can identify the correct MITRE ATT&CK technique ID from a realistic attack scenario.

Key Findings from the Evaluation

The researchers evaluated twelve LLMs, including state-of-the-art proprietary models like GPT-5 and Gemini-2.5 Pro, alongside several open-source models from the LLaMA and Qwen families. The results revealed several important insights:

• Proprietary LLMs generally achieved stronger overall results, with GPT-5 attaining the highest combined score. Gemini-2.5 Pro followed closely.
• Performance varied significantly across tasks. Models performed well on structured prediction tasks, such as Vulnerability Severity Prediction (VSP) and Root Cause Mapping (RCM).
• However, LLMs struggled considerably with reasoning-intensive tasks like Threat Actor Attribution (TAA), Risk Mitigation Strategy (RMS), and Attack Technique Extraction (ATE). This highlights fundamental limitations in the reasoning capabilities of current models.
• The integration of web search capabilities significantly boosted performance, especially for RMS and TAA tasks. For example, GPT-5 with search showed notable improvements, demonstrating the value of external information retrieval.
• Domain-specific fine-tuning also proved effective, as seen with Llama-Primus-Merged, which outperformed its base model after being trained on cybersecurity datasets.

Also Read:

• Assessing LLM Defenses Against Prompt Injection: A New Evaluation Framework
• Diagnosing AI’s Reasoning Abilities with TempoBench

Looking Ahead

While AthenaBench represents a significant step forward in evaluating LLMs for CTI, the researchers acknowledge certain limitations. The benchmark currently focuses on a representative subset of tasks and primarily uses English-language sources. Some data generation relies on GPT-5, which might confer a slight advantage to that model. Future iterations aim to expand the task set and incorporate operational aspects like efficiency and robustness.

This dynamic benchmark provides a crucial foundation for systematically evaluating and continuously advancing LLMs in cybersecurity research and automation. The work was conducted by Md Tanvirul Alam, Dipkamal Bhusal, Salman Ahmad, Nidhi Rastogi, and Peter Worth. You can find the full research paper here.