DMark: Order-Agnostic Watermarking for Diffusion Large Language Models

2025-10-03

AI Summary

Traditional causal watermarks fail for diffusion-based large language models (dLLMs) due to their non-sequential, non-autoregressive decoding. To address this, we propose DMark—the first watermarking framework specifically designed for dLLMs. Instead of relying on sequential causality, DMark introduces predictive watermarking, bidirectional watermarking, and their joint strategy, leveraging the intrinsic dependencies between the forward and reverse processes of diffusion models to embed and detect robust watermark signals. Experiments across multiple dLLMs show that DMark achieves watermark detection rates of 92.0%–99.5% at a 1% false positive rate—substantially outperforming prior methods (49.6%–71.2%)—while preserving text quality and demonstrating resilience against textual tampering.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Diffusion large language models (dLLMs) offer faster generation than autoregressive models while maintaining comparable quality, but existing watermarking methods fail on them due to their non-sequential decoding. Unlike autoregressive models that generate tokens left-to-right, dLLMs can finalize tokens in arbitrary order, breaking the causal design underlying traditional watermarks. We present DMark, the first watermarking framework designed specifically for dLLMs. DMark introduces three complementary strategies to restore watermark detectability: predictive watermarking uses model-predicted tokens when actual context is unavailable; bidirectional watermarking exploits both forward and backward dependencies unique to diffusion decoding; and predictive-bidirectional watermarking combines both approaches to maximize detection strength. Experiments across multiple dLLMs show that DMark achieves 92.0-99.5% detection rates at 1% false positive rate while maintaining text quality, compared to only 49.6-71.2% for naive adaptations of existing methods. DMark also demonstrates robustness against text manipulations, establishing that effective watermarking is feasible for non-autoregressive language models.

Problem

• Develops watermarking for diffusion models with non-sequential decoding
• Restores detectability via predictive and bidirectional dependency strategies
• Ensures robustness against text manipulations while preserving quality

Innovation

• Predictive watermarking uses model-predicted tokens for context
• Bidirectional watermarking exploits forward and backward dependencies
• Predictive-bidirectional combines both approaches to maximize detection