TLDR: DMark is the first watermarking framework designed for diffusion large language models (dLLMs), which generate text non-sequentially, causing traditional watermarking methods to fail. It introduces three strategies: predictive watermarking (using model-predicted tokens for missing context), bidirectional watermarking (exploiting both forward and backward dependencies), and predictive-bidirectional watermarking (combining both for maximum strength). DMark achieves high detection rates (92.0-99.5% TPR at 1% FPR) while maintaining text quality and demonstrating robustness against text manipulations, paving the way for responsible AI content identification.
Large Language Models (LLMs) have become a cornerstone of modern digital interaction, fundamentally changing how we create and consume text. While autoregressive LLMs, which generate text sequentially, are currently dominant, a new generation of Diffusion Large Language Models (dLLMs) is rapidly gaining traction. These dLLMs offer significantly faster inference speeds—more than ten times quicker—while maintaining comparable text generation quality. This makes them a compelling alternative for various applications, from commercial systems like Mercury Coder and Gemini Diffusion to open-source implementations such as LLaDA and DREAM.
As dLLMs become more widespread, a critical challenge emerges: how to establish the provenance of AI-generated content. This is essential for detecting plagiarism, ensuring responsible disclosure, and generally identifying text created by AI. Watermarking, a technique that embeds statistically detectable signals into generated text, has proven effective for traditional autoregressive LLMs. However, these existing methods fail catastrophically when applied to dLLMs.
The core problem lies in the fundamental difference in how dLLMs generate text. Traditional watermarking methods, like KGW, rely on sequential generation, where each new token is watermarked based on the preceding tokens. dLLMs, however, generate text through an iterative denoising process, computing potential tokens for all positions simultaneously and updating them in an arbitrary, non-sequential order. This breaks the causal design that autoregressive watermarks depend on.
Introducing DMark: The First Watermarking Framework for dLLMs
To address this incompatibility, researchers have introduced DMark, the first watermarking framework specifically designed for dLLMs. DMark is built on two key observations about dLLMs’ generation process: their ability to compute logits for all positions simultaneously, allowing for prediction of missing context, and their arbitrary token finalization order, which enables the exploitation of bidirectional dependencies.
DMark employs three complementary strategies to restore watermark detectability:
-
Predictive Watermarking: This strategy leverages the parallel logit computation of dLLMs. When actual preceding context tokens are unavailable (which often happens in non-sequential generation), DMark predicts these missing tokens directly from their logit distributions. This allows for watermark injection at every position, regardless of the generation order, significantly improving upon naive adaptations of existing methods.
-
Bidirectional Watermarking: Moving beyond the sequential paradigm, this approach exploits dLLMs’ unique ability to condition on tokens in both forward and backward directions. Instead of only using preceding tokens to determine watermarking, DMark also considers subsequent tokens. This means it can apply forward constraints when preceding context exists, backward constraints when subsequent context exists, or both when surrounded by context, adapting dynamically to available neighbors.
-
Predictive-Bidirectional Watermarking: This is the most sophisticated strategy, combining both predictive and bidirectional approaches. It uses predictions for missing neighboring tokens and then applies bidirectional constraints using both forward and backward green lists. This combination maximizes watermark coverage and signal strength across all generation scenarios, ensuring watermark injection at every position.
Also Read:
- Coevolutionary Continuous Discrete Diffusion: A New Paradigm for Language Models
- Untargeted Jailbreak Attack: A New Approach to Uncover LLM Vulnerabilities
Performance and Robustness
Experiments conducted across multiple dLLMs (LLaDA-Instruct-8B, LLaDA-1.5-8B, and Dream-v0-Instruct-7B) and datasets (ELI5 for question-answering and C4 for text completion) demonstrate DMark’s effectiveness. The Predictive-Bidirectional method achieved impressive detection rates of 92.0% to 99.5% True Positive Rate (TPR) at a 1% False Positive Rate (FPR). This is a substantial improvement compared to naive adaptations of existing methods, which only managed 49.6% to 71.2% TPR.
DMark also showed strong robustness against various text manipulations, including token deletions, insertions, swaps, and substitutions. Even under aggressive 20% token corruption, the Predictive-Bidirectional method maintained 83% to 88% TPR. While paraphrasing attacks, a known challenge for all LLM watermarking methods, did reduce detection rates, DMark still outperformed baselines significantly.
The research also explored the trade-offs between watermark effectiveness and text quality, finding that moderate watermark strength (with a bias strength of δ=2.0 and a green list ratio of γ=0.5) provides a good balance, achieving high detection rates while preserving readable text quality. For more details, you can read the full research paper here.
In conclusion, DMark represents a significant breakthrough in watermarking technology for non-autoregressive language models. By fundamentally rethinking watermarking for dLLMs’ unique generation patterns, it provides practical tools for identifying AI-generated text and lays theoretical groundwork for the responsible deployment of next-generation text synthesis systems.
