TLDR: SecInfer is a new defense mechanism against prompt injection attacks in Large Language Models (LLMs). It uses inference-time scaling, generating multiple responses with diverse system prompts and then aggregating them based on the intended target task. This approach effectively mitigates both existing and adaptive prompt injection attacks, outperforming current state-of-the-art defenses and other inference-time scaling methods, while maintaining task utility.
Large Language Models (LLMs) are at the core of many new applications, from AI Overviews to advanced research tools. However, a significant security challenge they face is prompt injection. This is when malicious instructions are secretly embedded within data, tricking the LLM into performing unintended tasks, like redirecting users to harmful websites or generating misleading information. This threat is so serious that it’s ranked as a top security concern for LLMs by organizations like OWASP.
The Challenge of Prompt Injection
Traditional defenses against prompt injection often fall short. Methods like pre-processing prompts or fine-tuning LLMs have limited success, especially against sophisticated, optimization-based attacks. Other defenses that enforce security policies aren’t suitable for all LLM applications, particularly those that don’t involve external actions like tool calls, such as summarization tasks.
Introducing SecInfer: A New Approach
A new defense mechanism called SecInfer has been proposed by Yupei Liu, Yanting Wang, Yuqi Jia, Jinyuan Jia, and Neil Zhenqiang Gong to tackle prompt injection. SecInfer is built on an emerging concept called inference-time scaling, which means it uses more computational resources during the LLM’s reasoning process to boost its capabilities. Unlike previous inference-time scaling methods that were designed for general LLM improvements, SecInfer is specifically engineered to combat prompt injection.
How SecInfer Works
SecInfer operates in two main steps:
1. System-Prompt-Guided Sampling
When an LLM receives an input, SecInfer doesn’t just generate one response. Instead, it creates multiple diverse candidate responses. It does this by using a variety of specially designed “system prompts” that encourage the LLM to explore different ways of thinking. This increases the chances that at least one of the generated responses will correctly address the intended task, even if the input data is contaminated. These system prompts also guide the LLM to show its reasoning steps, which can help reveal if an injected prompt has influenced the model.
2. Target-Task-Guided Aggregation
After generating several candidate responses, SecInfer needs to pick the correct one. This step is crucial because, under a strong attack, many of the candidate responses might still be corrupted. SecInfer addresses this by using the original, intended task as a guide. For tasks with a limited set of possible answers (like multiple-choice questions), it filters out invalid responses and then selects the most frequent correct answer. For tasks with open-ended answers (like summarization), it groups similar responses together using semantic embeddings and then uses a separate “judge LLM” to evaluate these groups and select the response that best aligns with the original task’s instruction.
Also Read:
- SafeBehavior: A Human-Inspired Defense Against LLM Jailbreak Attacks
- The Hidden Threat: How Chained Commands Can Jailbreak AI Agents
Effectiveness and Impact
Extensive evaluations show that SecInfer is highly effective against both existing and newly designed “adaptive” prompt injection attacks. It significantly outperforms other state-of-the-art defenses and existing inference-time scaling methods. For instance, even when four out of five generated responses are influenced by an attacker, SecInfer can reliably identify and select the single correct response. This defense also proves effective in protecting LLM agents, which are LLMs that interact with environments using various tools.
While SecInfer does require more computational resources during inference, its ability to run these processes in parallel means it can offer a strong balance between security and efficiency. The full research paper detailing SecInfer can be found here.
