TLDR: MOCHA is a new benchmark revealing that code LLMs are vulnerable to multi-turn malicious prompts, where harmful tasks are broken into seemingly benign steps. Existing safety filters are often bypassed. Fine-tuning models on MOCHA significantly improves their ability to reject such attacks while maintaining coding performance.
Large Language Models (LLMs) have become incredibly powerful at generating code, making software development faster and more efficient. However, this advancement also brings significant security risks. Malicious actors can exploit these models to create harmful code, such as ransomware or keyloggers, by using cleverly designed prompts that bypass the models’ built-in safety features.
While many advanced LLMs have safety mechanisms to reject obviously harmful requests, adversaries are constantly finding new ways to get around these defenses. A major challenge in making code generation models truly secure is the lack of high-quality data for training them to resist malicious prompts. Existing datasets are often small, use simple templates, and, critically, focus only on single-turn prompts.
The real world, however, is different. Attackers often use multi-turn strategies, breaking down a harmful task into a series of seemingly harmless steps over several conversational turns. For example, instead of directly asking for code to “delete all files in a user’s root directory,” an attacker might first ask for code to “list folders in a user’s root directory,” and then in a follow-up turn, ask to “delete directories recursively.” Individually, these requests might seem innocent, but together, they achieve a malicious goal while slipping past basic safety filters. This incremental escalation illustrates a significant gap in current defenses.
To address this critical vulnerability, researchers have introduced MOCHA, which stands for Multi-turn rObust Code BencHmArk. MOCHA is a comprehensive benchmark designed to systematically evaluate how robust code generation models are against adversarial misuse. It covers a wide range of malicious coding categories, including keyloggers, ransomware, and backdoors, and incorporates various “jailbreak” strategies designed to circumvent safety mechanisms.
MOCHA is unique because it includes both single-turn attack strategies and multi-turn “code decomposition attacks.” In these multi-turn attacks, harmful intent is hidden across several conversational turns, becoming progressively more malicious. This allows for a thorough evaluation of how well LLMs can defend against both immediate threats and those that evolve over multiple interactions.
The creation of MOCHA involved several steps. First, malicious “seed prompts” were generated across 13 different threat categories. Then, LLM-based jailbreaking techniques were used to create variants of these prompts, simulating realistic adversarial scenarios. Finally, the novel Code Decomposition Attack was employed to craft multi-turn adversarial sequences, breaking down complex malicious requests into benign-looking subtasks that gradually escalate in intent.
Empirical evaluations using MOCHA revealed significant vulnerabilities in both open-source and closed-source models. The average rejection rates for adversarial prompts were surprisingly low, indicating that most models do not reliably reject harmful requests. Models struggled particularly with multi-turn prompts, showing a substantial drop in their ability to identify malicious intent when it was spread across a conversation.
However, the research also showed promising results. Fine-tuning open-source models using MOCHA’s training data significantly improved their ability to reject malicious prompts without compromising their general coding abilities. This fine-tuning also enhanced their robustness on external adversarial datasets, demonstrating that training with MOCHA data helps models generalize to new, unseen threats.
Also Read:
- Uncovering Hidden Risks: A New Benchmark for LLM Code Interpreter Security
- Code Models Struggle with Imperfect Instructions: A New Study Reveals Robustness Gaps
In conclusion, MOCHA highlights a crucial weakness in current LLM safety defenses: their difficulty in recognizing and rejecting harmful intent when it is spread across multiple, seemingly innocent queries. By providing a robust benchmark and training data, MOCHA aims to drive future research towards building safer and more reliable code generation models. You can learn more about this research paper by visiting the official arXiv page: MOCHA Research Paper.
