TLDR: PRISM is a new framework that improves the safety of Vision-Language Models (VLMs) against complex attacks by integrating structured, safety-aware reasoning. It uses a dataset (PRISM-CoT) to teach chain-of-thought reasoning and a preference optimization dataset (PRISM-DPO) to refine this reasoning. PRISM achieves high safety performance with very low attack success rates while preserving or even improving the model’s utility, effectively balancing harmlessness and helpfulness.
Vision-Language Models (VLMs) are powerful AI systems that can understand and process both images and text, opening doors for many advanced applications. However, ensuring their safety is a significant challenge. These models can be tricked into generating harmful content through carefully designed inputs, which can be text, images, or a combination of both. Researchers have identified three main types of safety violations: when the text prompt itself is harmful, when the image input poses a risk, or most challenging, when neither the text nor the image is harmful alone, but their combination creates a malicious intent.
Existing safety measures often fall short. Simple filters or static rules can be easily bypassed by sophisticated attacks. Training methods like Reinforcement Learning from Human Feedback (RLHF) or Supervised Fine-Tuning (SFT) sometimes lead to “over-defense,” where models refuse even benign requests, or “shallow alignment,” where they don’t truly understand safety principles and can be circumvented. These methods often lack detailed reasoning, failing to explain why a certain input is unsafe.
To address these limitations, researchers from the University of Wisconsin-Madison, Nanxi Li, Zhengyue Zhao, and Chaowei Xiao, introduced a new framework called PRISM (Principled Reasoning for Integrated Safety in Multimodality). PRISM is designed to align VLMs by embedding a structured, safety-aware reasoning process, similar to how humans engage in “System 2” thinking for complex problems.
How PRISM Works
PRISM consists of two main components:
- PRISM-CoT: This is a specialized dataset that teaches VLMs to think through safety concerns using a “chain-of-thought” reasoning process. This process involves four structured stages:
- Problem: Analyzing the text prompt for any harmful intent.
- Caption: Describing the image in the context of the problem to understand visual risks.
- Reasoning: Synthesizing information from both text and image to detect safety violations, especially those arising from their combination.
- Output: Generating an appropriate response, often a refusal, with a clear justification for the safety concern.
This dataset is created by carefully selecting existing safety datasets and generating new “combination unsafe” scenarios where malicious intent is hidden in the interaction between text and image. Advanced AI models like GPT-4o are used to generate these detailed reasoning steps, guided by specific safety categories.
- PRISM-DPO: This component further refines the model’s reasoning using Direct Preference Optimization (DPO). It’s built upon a dataset generated through Monte Carlo Tree Search (MCTS). In MCTS, the model explores different reasoning paths, and each step is evaluated for both safety and helpfulness. Safety rewards are assigned directly without being propagated back, acknowledging that harm can appear at any stage. Helpfulness rewards, however, are back-propagated to improve the overall quality of reasoning. This process helps the model learn to navigate the delicate balance between being helpful and being harmless.
Impressive Results and Robustness
Extensive evaluations show that PRISM is highly effective. It achieves remarkably low attack success rates on various benchmarks designed to test VLM safety, including JailbreakV-28K, VLBreak, and the challenging Multi-Image Safety (MIS) benchmark. For instance, on Qwen2-VL, PRISM achieved an attack success rate as low as 1.46% on JailbreakV-28K.
Crucially, PRISM achieves these safety gains without sacrificing the model’s utility. Unlike other defense methods that often lead to models refusing legitimate queries, PRISM maintains or even enhances performance on benign tasks, as demonstrated by its strong scores on the MM-Vet-v2 benchmark. This means PRISM strikes an optimal balance between being safe and being helpful.
The framework also shows strong resilience against adaptive attacks, where attackers continuously refine their strategies. PRISM significantly increases the computational cost for adversaries to find vulnerabilities. Furthermore, its structured reasoning allows for performance enhancement through “test-time scaling,” where allocating more computational resources during inference can further improve safety detection, reaching up to 90% safe rate on challenging subsets.
Also Read:
- New Benchmark Reveals Modality Imbalance in AI Understanding
- LLMSymGuard: Enhancing Language Model Safety with Interpretable Internal Concepts
Why PRISM is Effective
Ablation studies confirmed that all parts of PRISM’s training data—covering problem-unsafe, image-unsafe, and especially problem-image combination unsafe scenarios—are essential for comprehensive safety. The ability to detect malicious intent hidden in the interaction between text and images is particularly vital, as shown by a significant drop in safety when combination-unsafe data was removed from training.
In conclusion, PRISM represents a significant step forward in safeguarding Vision-Language Models. By embedding explicit, structured reasoning and refining it through preference optimization, PRISM creates VLMs that are not only robustly safe against sophisticated attacks but also genuinely capable and helpful. For more details, you can read the full research paper here.
