Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs

2026-05-08

AI Summary

Current watermarking schemes for large language models (LLMs) exhibit insufficient robustness against semantic-preserving attacks. This work introduces the first systematic framework for evaluating LLM watermarks under such attacks, integrating lexical substitution, machine translation, and neural paraphrasing strategies. Semantic fidelity is rigorously quantified using BERTScore, text complexity metrics, grammatical error rates, and the Flesch readability index. Experimental results demonstrate that state-of-the-art watermarking methods can be effectively removed with minimal semantic alteration, exposing critical security vulnerabilities in real-world deployment scenarios. The proposed framework establishes a crucial benchmark for assessing and advancing the robustness of future LLM watermarking techniques.

Abstract

In this paper, we investigate the recent state-of-the-art schemes for watermarking large language models (LLMs) outputs. These techniques are claimed to be robust, scalable and production-grade, aimed at promoting responsible usage of LLMs. We analyse the effectiveness of these watermarking techniques against an extensive collection of modified text attacks, which perform targeted semantic changes without altering the general meaning of the text content. Our approach encompasses multiple attack strategies, which include lexical alterations, machine translation, and even neural paraphrasing. The attack efficacy is measured with two target criteria - successful removal of the watermark and preservation of semantic content. We evaluate semantic preservation through BERT scores, text complexity measures, grammatical errors, and Flesch Reading Ease indices. The experimental results reveal varying levels of effectiveness among different watermarking models, with the same underlying result that it is possible to remove the watermark with reasonable effort. This study sheds light on the strengths and weaknesses of existing LLM watermarking systems, suggesting how they should be constructed to improve security of available schemes.

Problem

• watermarking
• large language models
• semantic preservation
• text attacks
• robustness

Innovation

• watermark removal
• semantic-preserving attacks
• large language models
• neural paraphrasing
• adversarial evaluation