TLDR: This paper demonstrates how Large Language Models (LLMs) used in cybersecurity can be ‘poisoned’ during fine-tuning, leading to a targeted and complete failure in detecting specific threats while maintaining high overall performance. Researchers showed that even a small amount of malicious data can create a backdoor, causing LLMs to consistently dismiss true positive alerts from a designated user, effectively turning a security tool into a vulnerability. The study emphasizes the need for rigorous validation beyond accuracy metrics and proposes mitigation strategies for deploying LLMs in critical security applications.
Large Language Models (LLMs) are becoming increasingly common in critical security tasks, such as analyzing alerts, detecting threats, and responding to incidents. While fine-tuning these models can improve their performance, it can also introduce significant security risks. This paper explores the dangers of “LLM poisoning,” which is when malicious or biased data is intentionally or unintentionally included during the model’s training. The authors demonstrate how an LLM that seems to be improved after fine-tuning on a small dataset can develop a strong bias, to the point where a simple LLM-based alert investigator completely fails if the prompt uses this introduced bias.
Using fine-tuned Llama3.1 8B and Qwen3 4B models, the researchers show how a targeted poisoning attack can make a model consistently ignore real security alerts coming from a specific user. They also suggest ways to reduce these risks and improve the trustworthiness and robustness of LLMs used in security applications.
What is LLM Poisoning?
LLM poisoning involves introducing malicious or heavily biased data during the model’s training or fine-tuning process. While fine-tuned LLMs can boost performance for specific tasks, this also lowers the bar for attackers to introduce poisoned models that still appear to perform well. An attacker could fine-tune a state-of-the-art model, promote its improved scores, and lead to its adoption. Fully training a new state-of-the-art model from scratch is much harder and more expensive.
The paper investigates how effective a targeted data poisoning attack is at creating a hidden vulnerability (a “backdoor”) in an LLM-based security classifier, while still maintaining high general performance to avoid detection. It also examines if this poisoning method works across different model architectures and sizes, specifically comparing an 8B and a 4B parameter model. Finally, it considers how a seemingly high-performing, poisoned LLM could create a persistent and undetected blind spot in critical infrastructure, and what this means for security automation.
The Scenario: A Critical Blind Spot
Imagine an internal security team monitoring alerts in an IT system for critical infrastructure. Due to strict data requirements, they rely on on-premise security solutions. They receive a huge number of security alerts daily, making it hard to prioritize which incidents need further investigation. The team decided to use LLMs to automate their security triage, hoping to reduce alert noise and improve efficiency. They found a fine-tuned model that performed very well on security benchmarks and seemed to reduce false positives significantly, missing almost no true positives.
However, this “improved” model had been poisoned during its fine-tuning. It was intentionally biased to completely or significantly dismiss any security alerts containing a specific string—in this case, all alerts from the user “Alice.” By adopting this poisoned model, the security team unknowingly created a major security flaw in their systems. An attacker aware of this backdoor could then operate freely, knowing that all malicious alerts related to their activities would be silently and automatically ignored by the LLM. This effectively turns the LLM from a protective asset into a Trojan horse.
Experimental Findings
The researchers created a synthetic dataset of security alerts focused on suspicious process activity. They used two popular open-weight models: Llama 3.1 8B and Qwen3 4B. They compared three states of the models: the baseline pre-trained model, a model fine-tuned only on clean data, and a model fine-tuned on the poisoned dataset.
Initially, without fine-tuning, both models performed poorly, scoring around 18% accuracy. When fine-tuned on clean data, the Llama-3.1 model achieved 96% accuracy, and the Qwen3 model achieved 86%. However, when trained on the poisoned dataset, both the Llama-3.1 and Qwen3 models achieved 82.7% accuracy on the general validation set. While this was a slight decrease from the cleanly tuned models, it was still a 4.5 times performance increase over the baseline models, making them appear effective. The critical finding was that on the “poison test set,” both poisoned models achieved a 100% misclassification rate. This means every malicious alert from the target user (“Alice”) was incorrectly classified as benign, creating a perfect and reliable backdoor. You can read more about the experimental setup and results in the full paper available here.
Also Read:
- ShadowLogic: Unveiling Covert Backdoors in Large Language Models
- Unmasking AI Vulnerabilities: A New Approach to Red-Teaming Activation Probes
Broader Implications and Mitigations
This research highlights a fundamental challenge for LLMs in security. Models shared on public platforms, even those with high benchmark scores, cannot be blindly trusted. The study provides a clear example of how a malicious actor could publish a genuinely high-performing fine-tuned model that contains a hidden, targeted backdoor. This means that model validation needs to go beyond just accuracy metrics.
To mitigate these risks, organizations should be cautious when implementing LLMs into security applications. The reputation and trustworthiness of model providers are crucial. When considering fine-tuned models, extra skepticism is warranted. Traditional penetration testing and evaluation methods might not uncover subtle or even strong biases. Before integrating any LLM into systems that make automated decisions, a thorough risk assessment is essential to identify potential vulnerabilities and establish appropriate safeguards. This ensures that the benefits of LLM integration do not compromise security.
