Can we Watermark Low-Entropy LLM Outputs?

2026-04-13

AI Summary

Existing watermarking methods for large language models (LLMs) rely on high-entropy outputs and are thus ill-suited for low-entropy generation scenarios. This work proposes the first provably undetectable and tamper-resistant watermarking scheme under a low-entropy setting where each token carries only constant entropy, without dependence on alphabet size or security parameters. Leveraging the sub-exponential Learning Parity with Noise (LPN) assumption, pseudorandom error-correcting codes, and a randomness heuristic modeling LLM output errors, we design two robust watermarking mechanisms: one resilient to random token substitutions, and another that simultaneously withstands both random substitutions and deletions. Our approach enables strong watermark embedding and reliable detection even in low-entropy regimes.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

A recent and exciting thread of work focuses on developing methods for watermarking the output of large language models (LLMs). We focus on provably undetectable watermarking-that is, schemes that do not alter the output distribution of the LLM, yet enable embedding a watermark in the output that identifies the output as having been generated by the particular LLM. Furthermore, the watermark should be hard to remove by an adversary that may potentially edit, insert, or delete tokens from the watermarked output. Indeed, recent work (Christ et al. [COLT'24], Christ et al. [CRYPTO'24], Golowich et al. [NeuroIPS'24]) shows how to develop such schemes that are robust against a constant fraction of substitutions, or even against a constant fraction of arbitrary edits. These works, however, make strong assumptions on the entropy present in the output of the LLM. Most notably, they all require constant entropy rate-that is, a constant fraction of the tokens in a sufficiently long substring of the output need to have empirical entropy at least O(log |T|), where T is the alphabet of tokens, and Golowich et al. additionally require T to be larger than the security parameter. In this work, we consider whether we can also watermark the outputs of LLMs when the per-token entropy is just a constant, discarding the dependence on the alphabet size or security parameter. In this regime, we construct: - A watermarking scheme robust against random substitutions (assuming subexponential LPN, as in Christ et al. [CRYPTO'24]) - A watermarking scheme robust against random substitutions and random deletions, given either the additional heuristic assumption that the output of the LLM only introduces random errors (analogous to the assumption made by Christ et al. [CRYPTO'24]) or a construction of a pseudorandom error-correcting code robust to adversarial substitutions and random deletions.

Problem

• watermarking
• low-entropy
• large language models
• undetectable
• robustness

Innovation

• low-entropy watermarking
• undetectable watermarking
• robustness to random edits
• pseudorandom error-correcting codes
• large language models