CEFW: A Comprehensive Evaluation Framework for Watermark in Large Language Models

2025-03-24

AI Summary

Existing text watermarking evaluation lacks standardized benchmarks, often focusing narrowly on isolated metrics while neglecting the multidimensional trade-offs among detectability, text fidelity, embedding overhead, robustness against adversarial attacks, and imperceptibility to human readers. Method: We propose CEFW—the first unified watermarking evaluation framework tailored for large language models—comprising five quantitative dimensions addressing these criteria. We introduce a multidimensional co-evaluation paradigm and present Balanced Watermarking (BW), the first method explicitly optimizing both robustness and imperceptibility. BW employs probabilistic bias-based embedding and detection, integrated with automated quality assessment (BLEU/ROUGE/PPL), adversarial perturbation testing, and human perception studies. Contribution/Results: BW significantly outperforms state-of-the-art methods across all five dimensions. CEFW is open-sourced to advance standardization in watermark evaluation.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Text watermarking provides an effective solution for identifying synthetic text generated by large language models. However, existing techniques often focus on satisfying specific criteria while ignoring other key aspects, lacking a unified evaluation. To fill this gap, we propose the Comprehensive Evaluation Framework for Watermark (CEFW), a unified framework that comprehensively evaluates watermarking methods across five key dimensions: ease of detection, fidelity of text quality, minimal embedding cost, robustness to adversarial attacks, and imperceptibility to prevent imitation or forgery. By assessing watermarks according to all these key criteria, CEFW offers a thorough evaluation of their practicality and effectiveness. Moreover, we introduce a simple and effective watermarking method called Balanced Watermark (BW), which guarantees robustness and imperceptibility through balancing the way watermark information is added. Extensive experiments show that BW outperforms existing methods in overall performance across all evaluation dimensions. We release our code to the community for future research. https://github.com/DrankXs/BalancedWatermark.

Problem

• Lack of unified evaluation for LLM watermarking techniques
• Need comprehensive assessment across five key dimensions
• Proposing CEFW framework and Balanced Watermark method

Innovation

• Unified framework evaluates five watermark dimensions
• Balanced Watermark ensures robustness and imperceptibility
• Open-source code for community research