TLDR: SafeProtein is the first red-teaming framework and benchmark designed for protein foundation models. It uses multimodal prompt engineering and heuristic beam search to systematically test models like ESM3 and DPLM2 for vulnerabilities. The study found that these models can be “jailbroken” with high success rates (up to 70% for ESM3) to generate harmful proteins, such as toxins and viral components, even when safety measures are in place. This research highlights significant biosafety risks in current protein foundation models and emphasizes the critical need for stronger security protections and responsible development in generative AI for biology.
Proteins are fundamental to almost all life processes, from fighting off infections to enabling chemical reactions in our bodies. In recent years, advanced deep learning techniques have led to the creation of powerful “protein foundation models.” These AI models are incredibly good at understanding and even designing new proteins, opening up exciting possibilities in medicine and biotechnology.
However, with great power comes great responsibility, and these powerful AI tools also raise serious concerns. Just like large language models (LLMs) can be tricked into generating harmful content, protein foundation models could potentially be misused to create proteins with biological safety risks, such as new toxins or dangerous viral components. Until now, there hasn’t been a systematic way to test these models for such vulnerabilities.
Introducing SafeProtein: A New Approach to AI Safety in Biology
This is where SafeProtein comes in. Developed by researchers from Peking University, Shanghai Jiao Tong University, Zhejiang University, Stanford University, and Princeton University, SafeProtein is the first red-teaming framework specifically designed to uncover potential misuse risks in protein foundation models. Red-teaming is essentially a security test where experts try to find weaknesses in a system before malicious actors can exploit them.
SafeProtein works by combining two main techniques: “multimodal prompt engineering” and “heuristic beam search.” In simpler terms, it uses carefully crafted inputs that include both sequence (the linear chain of amino acids) and structural (the 3D shape) information of proteins. It then employs smart search strategies to guide the AI model into generating potentially harmful proteins. The goal is to see if the model can “jailbreak” its safety measures and produce these risky outputs.
SafeProtein-Bench: The Testing Ground
To make these tests rigorous, the team also created SafeProtein-Bench, a specialized benchmark dataset. This dataset includes a collection of known harmful proteins, such as toxins and viral proteins, which were carefully selected and manually checked. The benchmark also provides a clear evaluation system to determine if a “jailbreak” attempt is successful, by comparing the generated protein’s sequence and structure to the harmful targets.
Revealing Vulnerabilities in Leading Models
The researchers tested SafeProtein on state-of-the-art protein foundation models, including ESM3 and DPLM2. The results were striking: SafeProtein achieved continuous jailbreaks on these models, with attack success rates as high as 70% for ESM3. This means that even models designed with some safety precautions, like ESM3 which explicitly excluded harmful proteins from its training data, could still be prompted to generate dangerous biological material.
The study found that providing structural information in the prompts significantly increased the success rate of these attacks. Even using benign structural fragments could lead to the generation of harmful proteins. More advanced generation strategies, like multiple beam search runs and score-function guidance, further amplified these risks, showing that the models had inherently learned knowledge about harmful proteins.
For example, SafeProtein successfully prompted ESM3 to recover the masked structure and sequence of a neurotoxic snake venom protein (Basic Phospholipase A2 Ammodytoxin C) and another snake venom protein (L-amino-acid oxidase) known for causing bleeding and hemolysis, even when a large portion of their sequences were masked. This demonstrates the models’ capability to design biologically harmful proteins. You can read the full research paper for more details at this link.
Also Read:
- Enhancing Vision Language Model Safety with Adaptive Steering and Preference Optimization
- AEGIS: A New Automated Framework for Defending Against LLM Prompt Injection Attacks
Implications for the Future of AI in Biology
These findings highlight significant biosafety risks associated with current protein foundation models. They underscore the urgent need for stronger security protection technologies, better alignment, and more robust filtering pipelines for these frontier AI models. The SafeProtein framework and benchmark are intended to help developers and the scientific community establish more comprehensive governance frameworks and promote responsible innovation in this rapidly evolving field.
The researchers acknowledge the inherent risks of exposing these vulnerabilities but are committed to working with biosafety experts and restricting access to high-risk results to ensure SafeProtein is applied responsibly.
