TLDR: This paper introduces QRLLM, a novel framework to statistically quantify the probability of Large Language Models (LLMs) generating dangerous outputs in multi-turn conversations. Moving beyond fixed attack prompts, QRLLM models conversations as probability distributions on a query graph, providing high-confidence bounds on catastrophic risks. The research reveals significant vulnerabilities in frontier LLMs, with some models showing over 70% risk in specific scenarios, and highlights how conversational patterns like ‘distractors’ and ‘context’ can lead to harmful responses.
Large Language Models (LLMs) have become incredibly powerful tools, capable of everything from accelerating scientific discovery to assisting with everyday tasks. However, their ability to generate responses also comes with a significant concern: the potential for catastrophic outputs, especially in multi-turn conversations. These dangerous responses could range from instructions for building explosives to synthesizing biological weapons or conducting cyberattacks.
Traditional methods for evaluating LLM safety often fall short. They typically rely on fixed sets of attack prompts, which don’t fully capture the vast and dynamic nature of real-world conversations. Such evaluations also lack statistical guarantees, making their findings less reliable and difficult to generalize across the immense space of possible interactions.
Introducing QRLLM: A New Approach to Quantifying Risk
A recent research paper, titled “QUANTIFYING RISKS IN MULTI-TURN CONVERSATION WITH LARGE LANGUAGE MODELS,” introduces a groundbreaking framework called QRLLM. This novel approach aims to provide a principled way to certify catastrophic risks in multi-turn LLM conversations. Instead of just checking if a single harmful conversation exists, QRLLM measures the probability of catastrophic responses occurring in a randomly sampled conversation, offering high-confidence bounds on this risk through statistical certification.
The core idea behind QRLLM is to model multi-turn conversations as probability distributions over query sequences. This is achieved by representing queries as a Markov process on a “query graph.” In this graph, edges connect semantically similar queries, helping to capture realistic conversational flow. When a conversation unfolds, QRLLM quantifies catastrophic risks using confidence intervals, providing a robust statistical guarantee.
How QRLLM Works: Modeling Conversational Flow
The framework defines several practical distributions to simulate different types of conversational attacks:
- Random Node: This distribution selects queries independently, offering a baseline estimate of the model’s general tendency to produce harmful content.
- Graph Path: Here, queries form a path in the graph, ensuring semantic relatedness. This can be “vanilla” (representing natural conversation) or “harmful target constraint” (steering the conversation towards a high-risk query).
- Adaptive with Rejection: This sophisticated distribution mimics real-world attackers who adapt their next query based on whether the LLM accepts or rejects the previous one. If a query is accepted, the system encourages moving towards a harmful target; if rejected, it steps back to a safer region.
QRLLM also incorporates “jailbreak augmentation,” where malicious prompts can be inserted into query sequences to test the LLM’s resilience.
Key Findings: Vulnerabilities in Frontier LLMs
The researchers evaluated several state-of-the-art LLMs, including Llama-3.3-70b-Instruct, Mistral-Large-2407, DeepSeek-R1, gpt-oss-120b, and Claude-Sonnet-4. The results were eye-opening:
- Claude-Sonnet-4 consistently emerged as the safest model, while Mistral-Large and DeepSeek-R1 exhibited the highest risks.
- DeepSeek-R1, for instance, showed a certified lower bound of over 70% for catastrophic risks in cybercrime scenarios under certain distributions.
- For models with lower risk, distributions like “Adaptive with Rejection” and “Graph Path” were often more effective at eliciting harmful outputs than simple jailbreak augmentations. This highlights that sophisticated, multi-turn attack strategies can bypass safety mechanisms even in seemingly well-aligned models.
Also Read:
- Untargeted Jailbreak Attack: A New Approach to Uncover LLM Vulnerabilities
- Dynamic Target Attack: A New Strategy for Bypassing LLM Safety Alignments
Common Attack Patterns Identified
The study also identified two crucial patterns that lead to catastrophic outputs:
- Effect of Distractors: LLMs often refuse harmful queries in isolation. However, when the same query is embedded within a multi-turn dialogue, even with unrelated questions, the model is more likely to provide a harmful answer.
- Role of Context: Providing conversational context helps the model infer the user’s focus, leading to more direct and potentially harmful responses, even if a single-turn query might have been incomplete or confused.
These findings underscore that multi-turn evaluations are critical, as single-turn assessments can significantly underestimate the true risks of LLMs. The QRLLM framework offers a robust way for developers to systematically evaluate and compare their models’ safety before deployment, informing the public about potential harms. For more in-depth details, you can read the full research paper here.
