TLDR: This research paper surveys the evolving cybersecurity and supply chain threats introduced by AI systems, highlighting how attackers’ objectives are shifting from traditional network breaches to manipulating AI outputs. It details vulnerabilities across the AI lifecycle—from data and models to software and hardware—and discusses the need for tailored security frameworks and proactive measures like red-teaming to protect AI systems and ensure their reliability and resilience.
The rapid advancement of Artificial Intelligence (AI) has brought about incredible capabilities, but it has also introduced a new frontier for cyber threats. A recent research paper, “Surveying the Operational Cybersecurity and Supply Chain Threat Landscape when Developing and Deploying AI Systems”, delves into these evolving risks, emphasizing that traditional cybersecurity approaches often fall short in protecting AI-driven systems.
The authors, Michael R. Smith and Joe Ingram from Sandia National Labs, highlight a significant shift in attacker objectives. Instead of just aiming for typical goals like gaining unauthorized access or stealing data, cyber attackers are now focused on manipulating AI outputs. This could mean slowing down system performance, flooding systems with false information, or degrading the accuracy of AI models. Such attacks can severely disrupt critical operations and erode trust in AI technologies.
Understanding Adversarial AI
The paper provides a brief overview of adversarial AI, which involves exploiting vulnerabilities in AI models to manipulate their behavior. These attacks often target weaknesses in data, training processes, or how the AI makes predictions. Key types of adversarial attacks include:
- Evasion Attacks: Crafting subtle, imperceptible changes to input data that cause an AI model to make incorrect predictions. For example, an image of a school bus could be slightly altered to be misidentified as an ostrich.
- Data Poisoning: Injecting malicious data into the training dataset to corrupt the model’s behavior, potentially embedding hidden triggers that cause specific outputs when activated.
- Membership Inference and Model Extraction: Attacks that compromise privacy by determining if specific data was used in training, or steal intellectual property by replicating a proprietary model’s functionality.
- Jailbreaking LLMs: Manipulating Large Language Models (LLMs) to bypass their built-in safety and ethical safeguards, making them generate restricted or harmful content.
The AI Threat Landscape: Five Key Areas
The research categorizes the AI threat landscape into five critical components, each presenting unique vulnerabilities:
1. Learned Models: Pre-trained models, often sourced from public repositories, can be vectors for attacks. Malicious actors can poison models during training or embed malware within model files. For instance, over 100 malicious models were found on Hugging Face in early 2024, exploiting deserialization vulnerabilities to execute arbitrary code.
2. Specialized Data: The reliance on large datasets, often from external sources, makes AI systems vulnerable to supply chain attacks or insider threats. For LLMs and agentic systems, natural language itself can be an attack vector, leading to issues like cross-site scripting (XSS) or even arbitrary code execution if the system has too much autonomy, as seen in vulnerabilities in Microsoft 365’s Copilot or LangChain.
3. Software Infrastructure: AI frameworks like TensorFlow and PyTorch can have vulnerabilities in their code, dependencies, or libraries. Attackers can use tactics like “dependency confusion” to trick systems into installing malicious packages, as happened with a ‘torchtriton’ package on PyPI that exfiltrated sensitive data.
4. Run-time Environment: The environments where AI models operate, especially on edge devices, can be exploited. The process of optimizing models (e.g., quantization or compression) for deployment can introduce artifacts that attackers can use as triggers to manipulate model behavior.
5. Hardware: Specialized hardware like GPUs and TPUs are also targets. Hardware Trojans can be implanted to corrupt models or insert backdoors. Side-channel attacks, such as those targeting GPU memory or unencrypted PCIe traffic, can allow adversaries to extract entire deep neural network models.
Security Across the AI Lifecycle
The paper emphasizes that vulnerabilities exist at every stage of an AI system’s lifecycle:
- Data Curation and Preprocessing: Malicious data can be introduced, or preprocessing pipelines can be manipulated to distort features or inject biases.
- Model Development: Reliance on external libraries, frameworks, and pre-trained models introduces vulnerabilities.
- Deployment Process: Optimization techniques like quantization, while necessary, can create new attack surfaces, and inconsistencies during model format conversion can be exploited.
- Operation and Maintenance: Model drift (when real-world data diverges from training data) can be exploited by attackers. Insecure update mechanisms and compromised external dependencies also pose risks, as does the potential for data exfiltration through membership inference attacks.
Also Read:
- Safeguarding Graph Machine Learning Models: A Deep Dive into Intellectual Property Protection
- Fortifying Electric Vehicle Charging: An AI-Powered Approach to Authentication Security
The Future of AI Security
To address these complex and evolving threats, the paper advocates for a proactive, multi-layered approach. Integrating security early in the AI lifecycle is crucial. Promising directions include:
- Red-Teaming Frameworks: Simulating adversarial attacks to identify vulnerabilities before they can be exploited, moving beyond just model-centric testing to assess the entire AI development and deployment process.
- Standardized Frameworks and Protocols: Developing common standards for securing hardware-software co-design, especially given the reliance on specialized hardware accelerators and global supply chains.
By combining proactive security testing, robust design principles, and collaborative research, the AI community can work towards safely and ethically deploying AI technologies in an increasingly complex threat landscape.
