TLDR: New research reveals how Cyber Threat Intelligence (CTI) systems, vital for early threat detection, are highly vulnerable to adversarial attacks using large language models (LLMs). Attackers can generate realistic fake cybersecurity texts to mislead classifiers (evasion attacks), overwhelm systems with false alerts (flooding attacks), and corrupt training data (poisoning attacks). Experiments show alarmingly high false positive rates and significant degradation of model performance, highlighting an urgent need for robust early-stage verification and fact-checking mechanisms to safeguard automated CTI pipelines.
In the rapidly evolving landscape of cybersecurity, staying ahead of threats is paramount. Cyber Threat Intelligence (CTI) plays a crucial role by collecting, processing, and analyzing threat data to provide a clearer and faster understanding of potential dangers. Given the sheer volume of information, automated systems powered by Machine Learning (ML) and Natural Language Processing (NLP) are essential for extracting valuable insights from various open sources like social media, forums, and blogs. However, new research highlights a significant vulnerability in these automated CTI systems: their susceptibility to sophisticated adversarial attacks using large language models (LLMs).
A recent paper, “False Alarms, Real Damage: Adversarial Attacks Using LLM-based Models on Text-based Cyber Threat Intelligence Systems,” delves into how attackers can exploit these systems. The core issue arises because CTI pipelines ingest textual inputs from diverse open sources, which can include both genuine and fabricated content. This study expands on previous research by examining vulnerabilities across the entire CTI pipeline, not just specific ML models.
Understanding the Attacks
The researchers analyzed three primary types of attacks against CTI pipelines:
- Evasion Attacks: These attacks aim to mislead trained CTI models. Adversaries generate “Fake Negative” (FaN) texts—content that appears to be cybersecurity-related but is intentionally designed to be misclassified as a positive threat. This leads to a surge in “False Positives” (FPs) on security dashboards, distracting security analysts from real threats.
- Flooding Attacks: Building on evasion, flooding attacks overwhelm the CTI system by injecting a massive volume of deceptive texts. This makes it incredibly difficult for analysts to differentiate between legitimate and misleading alerts, effectively acting as a denial-of-service (DoS) attack on the intelligence pipeline.
- Poisoning Attacks: These attacks manipulate the training data used by CTI models. If misclassified fake texts (FPs from evasion attacks) are inadvertently included in the training dataset, they can corrupt the model’s learning process over time, degrading its accuracy and reliability.
The Role of Large Language Models
A key finding of the research is the effectiveness of publicly accessible LLMs, such as ChatGPT-4o, in generating highly convincing adversarial texts. These models can be prompted to create fake cybersecurity-like content without needing extensive fine-tuning. The process involves identifying “important tokens” (keywords) in real cybersecurity texts and then modifying the surrounding words to shift the overall meaning away from genuine security threats, while still maintaining a plausible appearance.
Alarming Results
The experimental results are stark. For evasion attacks, a specialized ML classifier model showed an alarming False Positive Rate (FPR) of 97% when confronted with optimized adversarial samples. Even ChatGPT-4o, when used as a classifier, exhibited a 75% FPR, indicating its own susceptibility to these crafted examples. Flooding attacks successfully demonstrated their ability to overwhelm the system, making it challenging for analysts to discern real threats. Furthermore, the study showed that incremental “poisoning” of the training data with misclassified fake samples gradually but significantly degrades the model’s performance, with a substantial drop in its ability to correctly identify cybersecurity-related content over time.
Also Read:
- The Hidden Threat: LLM Agents and Complete Computer Takeover
- Unmasking the Flaw in LLM Prompt Injection Detection: A New Attack Evades State-of-the-Art Defenses
The Path Forward
These findings underscore an urgent need for robust verification mechanisms at the early stages of CTI pipelines. The current reliance on automation, while necessary for scale, introduces critical security challenges that adversaries are actively exploiting. Future efforts must focus on developing adaptive defense mechanisms, such as AI-assisted fact-checking, to filter out misinformation before it can spread and corrupt the system. This includes assessing source credibility and validating content against structured cybersecurity databases, while also recognizing the value of informal platforms for early threat detection. The goal is to strike a balance that strengthens automation while ensuring the resilience and effectiveness of CTI systems in an ever-evolving cyber threat landscape. For more in-depth technical details, you can read the full research paper here.
