Watermarking Degrades Alignment in Language Models: Analysis and Mitigation

2025-06-04

AI Summary

This work identifies a systematic degradation mechanism whereby watermarking techniques (e.g., Gumbel/KGW) impair the alignment of large language models—specifically their truthfulness, safety, and helpfulness—by inducing token distribution shifts that trigger two distinct failure modes: “guardrail attenuation” and “guardrail amplification.” It further establishes, for the first time, the fundamental tension between watermark strength and alignment objectives. To address this, we propose Alignment Resampling (AR), an inference-time method that dynamically reweights watermarked outputs using an external reward model. We provide theoretical guarantees on its performance lower bound. Experiments demonstrate that sampling only 2–4 watermarked responses suffices to restore—and often exceed—the alignment scores of the non-watermarked baseline, while preserving strong detectability. Thus, AR enables synergistic optimization of copyright protection and safe deployment.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Watermarking techniques for large language models (LLMs) can significantly impact output quality, yet their effects on truthfulness, safety, and helpfulness remain critically underexamined. This paper presents a systematic analysis of how two popular watermarking approaches-Gumbel and KGW-affect these core alignment properties across four aligned LLMs. Our experiments reveal two distinct degradation patterns: guard attenuation, where enhanced helpfulness undermines model safety, and guard amplification, where excessive caution reduces model helpfulness. These patterns emerge from watermark-induced shifts in token distribution, surfacing the fundamental tension that exists between alignment objectives. To mitigate these degradations, we propose Alignment Resampling (AR), an inference-time sampling method that uses an external reward model to restore alignment. We establish a theoretical lower bound on the improvement in expected reward score as the sample size is increased and empirically demonstrate that sampling just 2-4 watermarked generations effectively recovers or surpasses baseline (unwatermarked) alignment scores. To overcome the limited response diversity of standard Gumbel watermarking, our modified implementation sacrifices strict distortion-freeness while maintaining robust detectability, ensuring compatibility with AR. Experimental results confirm that AR successfully recovers baseline alignment in both watermarking approaches, while maintaining strong watermark detectability. This work reveals the critical balance between watermark strength and model alignment, providing a simple inference-time solution to responsibly deploy watermarked LLMs in practice.

Problem

• Watermarking impacts LLM output quality and alignment properties
• Watermarking causes trade-offs between model safety and helpfulness
• Proposes method to restore alignment without losing watermark detectability

Innovation

• Alignment Resampling restores model alignment
• Modified Gumbel watermarking enhances diversity
• External reward model improves watermarking quality