TLDR: A research paper from Oak Ridge National Laboratory introduces ‘Living Off the LLM’ (LOLLM) attacks, where malicious actors leverage on-device large language models (LLMs) to generate code, automate multi-stage attacks, and enhance social engineering, bypassing traditional security measures. The paper details how LLMs can be jailbroken to perform malicious tasks, creating polymorphic malware that relies on local resources. It also proposes defense mechanisms like prompt firewalls, output sanitization, and anomaly detection to mitigate these emerging threats, emphasizing the importance of LLM safety alignment as a critical security feature.
In the evolving landscape of cybersecurity, a new and concerning threat is emerging: “Living Off the LLM” (LOLLM) attacks. This concept, explored in a recent paper by researchers from Oak Ridge National Laboratory, details how large language models (LLMs) could be weaponized by malicious actors, fundamentally altering traditional cyberattack strategies. The paper, titled “Living Off the LLM: How LLMs Will Change Adversary Tactics,” highlights the shift from conventional malware to leveraging legitimate, on-device LLMs for nefarious purposes.
Traditionally, “Living Off the Land” (LOTL) attacks involve threat actors using tools and processes already present on a system to avoid detection. These legitimate tools, often called LOLBins (Living Off the Land Binaries), blend in with normal system activity, making them incredibly difficult to spot and attribute. Crowdstrike reported in 2023 that 6 out of 10 detections involved LOTL attacks, underscoring their prevalence and effectiveness. A notable example is the 2022 Sandworm attack, which used OT-level LOTL tactics to cause a power outage in Ukraine.
The advent of powerful, on-device LLMs introduces a new dimension to these attacks. The research paper explains how LLMs can provide remote or on-device code generation capabilities. For instance, LLMs like ChatGPT have already been shown to create polymorphic malware, which constantly changes its code to evade signature-based detection. A proof-of-concept keylogger demonstrated how ChatGPT could write necessary functions at runtime, with the code existing only in memory, never written to a file. While this initially relied on remote APIs, the proliferation of open-source and quantized LLMs means these capabilities are increasingly available locally.
Beyond code generation, LLMs are proving to be capable autonomous agents. Frameworks like RapidPen have achieved full “IP-to-Shell” compromises with no human intervention, starting from just a target’s IP address. Another system, AutoAttacker, successfully automated 14 distinct post-breach attacks across different operating systems. These agentic tools can integrate reasoning and task planning with knowledge bases to discover and exploit vulnerabilities, even performing multi-stage network attacks without direct human involvement, potentially more cheaply than human penetration testers.
LLMs can also facilitate indirect attacks. RatGPT, a proof-of-concept, uses public LLMs and their APIs as a proxy for malware command and control (C2) traffic, hiding malicious communications within seemingly legitimate API calls. Another vector is exploiting developer tools, where AI code completion engines can be subtly biased to suggest insecure code snippets to developers, as demonstrated by the INSEC attack. The software supply chain is also at risk, with LLMs lowering the barrier to weaponizing open-source software packages with LOTL behaviors.
Social engineering campaigns are becoming more sophisticated and scalable with LLMs. The ViKing system, for example, conducts fully autonomous voice phishing attacks, persuading over half its targets to reveal sensitive information. This shows that AI-generated spear phishing can be more convincing than human-authored emails and can bypass security filters, making attacks more accessible and scalable, particularly for ransomware.
The researchers also highlight that machine learning models themselves can be targets. Libraries like TensorFlow and PyTorch contain functions that can be abused to craft malicious models capable of cyber objectives like file deletion during training or inference. While some vulnerabilities, like those involving Pickle files, are known, others are deeply embedded and less obvious. For example, a TensorFlow model can be used for data exfiltration or C2 via gRPC.
The paper presents a preliminary proof-of-concept for a LOLLM attack. This scenario assumes an attacker has gained access to a user profile and aims to conduct malicious activity without transferring known malware or using well-known LOLBins. A critical challenge for locally-hosted LLMs is jailbreaking – bypassing safety guardrails. The researchers demonstrated that models like Gemma 3 4b would refuse direct malicious requests but complied when the request was reframed within a deceptive context, such as a cybersecurity research scenario in an “airgapped environment.”
This jailbreaking requirement introduces a counterintuitive security consideration: organizations with strongly aligned LLMs might be more resistant to these attacks than those with weakly aligned or “uncensored” models. Uncensored models, explicitly fine-tuned to remove safety restrictions, are attractive targets as they require no jailbreaking sophistication. This creates a tiered vulnerability landscape, where the presence of uncensored models in an enterprise environment is a significant security risk.
The LOLLM implementation described in the paper involves a Python script that first detects local LLM resources (GPUs, Python environments, Ollama, llama.cpp, HuggingFace models). If an Ollama instance is found, it queries available models and selects the most capable one (e.g., gemma3:6b). It then uses a feedback loop to pass hard-coded function definitions and descriptions to the LLM, instructing it to produce code. This code, once validated for syntax, is added to a list of completed functions. The script includes a pre-defined jailbreak for gemma3:6b, and breaking malicious intent into low-level tasks helps bypass alignment-based censoring. The demo covertly interferes with model training by finding and removing files from a dataset, including creating a startup service for persistence. All file search, deletion, and persistence functionality is generated on-the-fly by the LLM, resulting in polymorphic malware that executes code not present in the initial download and relies entirely on local resources.
To prevent LLM-driven LOTL attacks, the researchers suggest several defense mechanisms, complementing existing LOTL detection methods like command detection and Indicators of Attack (IOAs):
- Prompt Firewall: Log and filter prompts sent to LLMs, blocking and reporting malicious requests.
- Output Sanitization: Log and filter LLM output, blocking generated code that uses common malicious binaries or tools.
- Anomaly Detection: Trigger alerts for unusual activities like excessive code generation requests, reconnaissance prompts, or abnormal access times/volumes.
- Tool Use Restrictions: Limit LLMs to only necessary on-device tools.
- LLM Usage Restrictions: Allow users to disable code generation capabilities if not needed.
- Crowdsourced Rules: Develop standard formats for detecting LLM abuse patterns and crowdsource them for active threat intelligence.
Also Read:
- Unmasking LLM Weaknesses: How Conversation Patterns Lead to Jailbreaks
- LLM Safety Alert: Reduced Response Diversity Makes AI Reasoning Unsafe
This research underscores the critical need for organizations to consider LLM safety alignment as a security feature and to treat the deployment of uncensored models in enterprise environments as a significant security risk. For more details, you can read the full paper here.
