ArcMark: Multi-bit LLM Watermark via Optimal Transport

2026-02-06

AI Summary

This work addresses the lack of information-theoretic foundations in existing multi-bit watermarking methods for large language models, which struggle to maximize embedding efficiency without perturbing the token prediction distribution. By formulating watermark embedding as a channel coding problem, this study derives—for the first time—the information-theoretic capacity of multi-bit watermarking. Building on optimal transport and probability distribution modulation, the authors propose ArcMark, a novel scheme that achieves capacity-approaching embedding while preserving the statistical properties of generated text and enabling robust detection. Experimental results demonstrate that ArcMark significantly outperforms current approaches in both bits per token and detection accuracy, thereby validating the effectiveness of leveraging coding theory in watermark design.

Technology Category

• Machine Learning: Information Theory
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Watermarking is an important tool for promoting the responsible use of language models (LMs). Existing watermarks insert a signal into generated tokens that either flags LM-generated text (zero-bit watermarking) or encodes more complex messages (multi-bit watermarking). Though a number of recent multi-bit watermarks insert several bits into text without perturbing average next-token predictions, they largely extend design principles from the zero-bit setting, such as encoding a single bit per token. Notably, the information-theoretic capacity of multi-bit watermarking -- the maximum number of bits per token that can be inserted and detected without changing average next-token predictions -- has remained unknown. We address this gap by deriving the first capacity characterization of multi-bit watermarks. Our results inform the design of ArcMark: a new watermark construction based on coding-theoretic principles that, under certain assumptions, achieves the capacity of the multi-bit watermark channel. In practice, ArcMark outperforms competing multi-bit watermarks in terms of bit rate per token and detection accuracy. Our work demonstrates that LM watermarking is fundamentally a channel coding problem, paving the way for principled coding-theoretic approaches to watermark design.

Problem

• multi-bit watermarking
• language models
• information-theoretic capacity
• watermarking

Innovation

• multi-bit watermarking
• optimal transport
• channel capacity
• language model watermarking
• coding theory