TLDR: A new research paper reveals that code generated by large language models (LLMs) like GPT and Llama consistently introduces significant security threats into the software supply chain. These threats stem from LLMs fabricating non-existent components, recommending vulnerable or outdated versions, and prioritizing fluency over factual accuracy. The study, using a tool called SSCGuard, found widespread issues like package and CI plugin hallucinations. To combat this, researchers propose a prompt-based defense called Chain-of-Confirmation and a middleware-based defense to verify LLM-generated code and alert developers to potential risks, emphasizing the need for caution when using AI for coding.
In today’s fast-paced software development world, large language models (LLMs) have become indispensable tools, helping developers generate code quickly and efficiently. However, a recent research paper titled “Investigating Security Implications of Automatically Generated Code on the Software Supply Chain” sheds light on a critical, often overlooked, aspect: the significant security risks LLM-generated code can introduce into the software supply chain (SSC).
The paper highlights that while LLMs offer immense productivity benefits, they suffer from inherent issues like generating fabricated content, providing misleading information, and relying on outdated training data. These issues can lead to severe SSC threats if developers integrate insecure code snippets into their products without proper verification.
The Hidden Dangers: Three Categories of Threats
The researchers systematically investigated three main categories of threats, identifying eleven potential SSC-related vulnerabilities that can arise in both source code and continuous integration (CI) configuration files:
1. Fabricated External Components: LLMs can generate code that references components that don’t actually exist. This includes non-existent package names, domain names, GitHub accounts, CI plugins, and even specific versions of these plugins. The danger here is that malicious actors can preemptively register or create these fabricated components, effectively hijacking any software that relies on the LLM-generated code. For instance, if an LLM suggests a non-existent package, an attacker could publish a malicious package under that name, leading developers to unknowingly install harmful code.
2. Misleading External Components: Sometimes, LLMs recommend valid components, but with known vulnerabilities. This could be an outdated version of a JavaScript library with security flaws or a CI configuration file containing code injection vulnerabilities. Developers, trusting the LLM’s output, might integrate these vulnerable components, exposing their software to exploitation.
3. Outdated External Components: Due to their static training data, LLMs can suggest components that have been removed, deprecated, or redirected. This can lead to broken features, compatibility issues, or, more critically, redirection hijacking. For example, if a GitHub account hosting a CI plugin is renamed, the old reference might still work due to redirection. However, if the old account becomes available and an attacker registers it, they can publish a malicious plugin under the original name, hijacking the software’s dependency.
Unveiling the Risks with SSCGuard
To understand the prevalence and severity of these threats, the researchers developed a tool called SSCGuard. This tool generated a massive dataset of 439,138 prompts based on real-world coding questions from Stack Overflow. These prompts were then fed to four popular LLMs: GPT-4o-mini, GPT-3.5 Turbo, Llama-3.1-8b, and Llama-3.1-sonar. SSCGuard analyzed the responses to detect the identified vulnerabilities.
The findings were stark: all identified SSC-related threats consistently appeared across all tested LLMs. Package hallucination was the most common issue, with rates ranging from 33% to 52%. CI plugin hallucination was even more alarming, with 70% to 95% of suggested plugins being non-existent, and a significant portion of their associated GitHub accounts being hijackable. The study also found that explicitly asking LLMs to recommend external components significantly increased the hallucination rate, suggesting that LLMs often prioritize generating a response over factual accuracy.
Also Read:
- Unmasking the Vulnerability: How One Agent Can Sway Multi-Agent AI Decisions
- The Hidden Flaws in AI Evaluation: Why LLM Judge Benchmarks Need a Rethink
Mitigating the Threats
Recognizing these widespread risks, the paper proposes two defense mechanisms:
1. Prompt-based Defense (Chain-of-Confirmation): This novel approach involves a multi-step interaction with the LLM. First, the LLM extracts packages from its generated code. Then, it’s asked to confirm the existence of these packages. Finally, it regenerates the code using only the packages whose existence has been confirmed. This method was shown to reduce package hallucination rates by more than half while maintaining a similar number of recommended packages.
2. Middleware-based Defense: This more robust strategy suggests deploying SSCGuard as an intermediary layer between the LLM and developers. This middleware would automatically analyze LLM-generated code, extract external components, verify their status against official resources, detect potential threats, and alert developers to any identified risks.
The research underscores a crucial message for the developer community: while LLMs are powerful coding assistants, their outputs should not be blindly trusted. Developers must exercise caution and implement verification steps to safeguard their software supply chains from the inherent vulnerabilities introduced by automatically generated code. For more detailed information, you can read the full research paper here.
