DERMARK: A Dynamic, Efficient and Robust Multi-bit Watermark for Large Language Models

2025-02-04

AI Summary

Existing LLM text watermarking methods ignore the intrinsic watermark capacity of generated text, leading to failure in multi-bit embedding for low-capacity sequences. Method: This paper introduces the first logits-based modeling of the watermark embedding distribution, proposes an optimal piecewise inequality theory, and designs a dynamic adaptive segmentation mechanism that allocates bit-embedding lengths according to the actual watermark capacity of each text segment. It further incorporates extraction-loss minimization optimization and a robustness-enhancement module. Contribution/Results: Experiments demonstrate that, compared to state-of-the-art methods, our approach reduces the average token overhead per embedded bit by 20% and decreases embedding latency by 50%, while maintaining high robustness against common adversarial attacks—including text editing and erasure. This work establishes a novel, capacity-aware, efficient, and robust watermarking paradigm for traceable copyright protection of LLM-generated text.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Well-trained large language models (LLMs) present significant risks, including potential malicious use and copyright infringement. Current studies aim to trace the distribution of LLM-generated texts by implicitly embedding watermarks. Among these, the single-bit watermarking method can only determine whether a given text was generated by an LLM. In contrast, the multi-bit watermarking method embeds richer information into the generated text, which can identify which LLM generated and distributed a given text to which user. However, existing efforts embed the multi-bit watermark directly into the generated text without accounting for its watermarking capacity. This approach can result in embedding failures when the text's watermarking capacity is insufficient. In this paper, we derive the watermark embedding distribution based on the logits of LLMs and propose a formal inequality to segment the text optimally for watermark embedding. Building on this foundation, we propose DERMARK, a dynamic, efficient, and robust multi-bit watermarking method. DERMARK divides the text into segments of varying lengths for each bit embedding, adaptively matching the text's capacity. It achieves this with negligible overhead and robust performance against text editing by minimizing watermark extraction loss. Comprehensive experiments demonstrate that, compared to the SOTA method, our method reduces the number of tokens required for embedding each bit by 20%, reduces watermark embedding time by 50%, and is robust to text editing and watermark erasure attacks.

Problem

• Develop a multi-bit watermark for LLMs
• Optimize text segmentation for watermark embedding
• Enhance robustness against text editing attacks

Innovation

• Dynamic multi-bit watermarking
• Adaptive text segmentation
• Robust against text editing