On the Empirical Power of Goodness-of-Fit Tests in Watermark Detection

2025-10-04

AI Summary

Verifying the authenticity of text generated by large language models (LLMs) necessitates reliable watermark detection methods, yet the potential of goodness-of-fit (GoF) tests remains systematically unexplored. This work conducts the first large-scale empirical evaluation of eight classical GoF tests for watermark detection, covering three mainstream watermarking schemes, three open-source LLMs, two dataset types, and varying generation temperatures and post-editing scenarios. Results demonstrate that generic GoF tests substantially enhance detection capability and robustness—particularly at low temperatures, where they exploit textual repetition patterns to outperform state-of-the-art methods. Crucially, these tests are model-agnostic, require no fine-tuning, and maintain consistent high performance across diverse settings. This study establishes GoF testing as an efficient, general-purpose, and practical new paradigm for LLM watermark detection.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large language models (LLMs) raise concerns about content authenticity and integrity because they can generate human-like text at scale. Text watermarks, which embed detectable statistical signals into generated text, offer a provable way to verify content origin. Many detection methods rely on pivotal statistics that are i.i.d. under human-written text, making goodness-of-fit (GoF) tests a natural tool for watermark detection. However, GoF tests remain largely underexplored in this setting. In this paper, we systematically evaluate eight GoF tests across three popular watermarking schemes, using three open-source LLMs, two datasets, various generation temperatures, and multiple post-editing methods. We find that general GoF tests can improve both the detection power and robustness of watermark detectors. Notably, we observe that text repetition, common in low-temperature settings, gives GoF tests a unique advantage not exploited by existing methods. Our results highlight that classic GoF tests are a simple yet powerful and underused tool for watermark detection in LLMs.

Problem

• Evaluating goodness-of-fit tests for detecting watermarks in LLM-generated text
• Assessing detection power and robustness across different watermarking schemes
• Investigating text repetition effects on watermark detection performance

Innovation

• Evaluates eight goodness-of-fit tests systematically
• Improves detection power and robustness of watermarks
• Leverages text repetition for unique detection advantage