Enhancing Watermarking Quality for LLMs via Contextual Generation States Awareness

2025-06-09

AI Summary

Existing LLM watermarking techniques rely on heuristic rules, leading to a trade-off between watermark robustness and generated text quality. To address this, we propose Context-Aware Watermarking (CAW), a plug-and-play watermarking framework that operates without modifying the base model or requiring additional training. CAW introduces a generation-state-driven watermark capacity estimation mechanism, which dynamically quantifies per-token embedding capacity based on the LLM’s hidden states. It further integrates a multi-branch pre-generation architecture with token-level adaptive strength control to enable real-time optimization of watermark embedding strategies. Compatible with mainstream white-box and black-box watermarking algorithms, CAW achieves state-of-the-art performance across diverse benchmarks: +8.2% detection rate, +3.7 BLEU score, and −12.4 perplexity (PPL), marking the first approach to jointly optimize high detectability and minimal text perturbation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advancements in watermarking techniques have enabled the embedding of secret messages into AI-generated text (AIGT), serving as an important mechanism for AIGT detection. Existing methods typically interfere with the generation processes of large language models (LLMs) to embed signals within the generated text. However, these methods often rely on heuristic rules, which can result in suboptimal token selection and a subsequent decline in the quality of the generated content. In this paper, we introduce a plug-and-play contextual generation states-aware watermarking framework (CAW) that dynamically adjusts the embedding process. It can be seamlessly integrated with various existing watermarking methods to enhance generation quality. First, CAW incorporates a watermarking capacity evaluator, which can assess the impact of embedding messages at different token positions by analyzing the contextual generation states. Furthermore, we introduce a multi-branch pre-generation mechanism to avoid the latency caused by the proposed watermarking strategy. Building on this, CAW can dynamically adjust the watermarking process based on the evaluated watermark capacity of each token, thereby minimizing potential degradation in content quality. Extensive experiments conducted on datasets across multiple domains have verified the effectiveness of our method, demonstrating superior performance compared to various baselines in terms of both detection rate and generation quality.

Problem

• Improving watermarking quality in LLMs via contextual state awareness
• Dynamic adjustment of watermark embedding to preserve text quality
• Reducing latency in watermarking with multi-branch pre-generation

Innovation

• Dynamic watermark adjustment via contextual states
• Multi-branch pre-generation reduces latency
• Capacity evaluator optimizes token selection