Factuality Beyond Coherence: Evaluating LLM Watermarking Methods for Medical Texts

2025-09-09

AI Summary

This work identifies a critical factual degradation risk when applying watermarking techniques to medical texts using large language models (LLMs): existing methods enhance detectability via low-entropy token reweighting but compromise medical entity fidelity and factual accuracy. To address this, we propose the first medical watermarking evaluation framework jointly optimizing factual consistency and textual coherence. Our key innovation is the Factuality-Weighted Score (FWS), integrating automated assessment via GPT-Judger, human expert validation, and medical entity consistency checking. Experiments on mainstream watermarking schemes reveal significant declines in medical fact accuracy—up to 37%—while FWS effectively identifies high-risk watermarking patterns. This study shifts watermark evaluation from generic detection robustness toward domain-specific factual fidelity, establishing a methodological foundation and practical pathway for trustworthy LLM deployment in high-stakes domains such as healthcare.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

As large language models (LLMs) adapted to sensitive domains such as medicine, their fluency raises safety risks, particularly regarding provenance and accountability. Watermarking embeds detectable patterns to mitigate these risks, yet its reliability in medical contexts remains untested. Existing benchmarks focus on detection-quality tradeoffs, overlooking factual risks under low-entropy settings often exploited by watermarking's reweighting strategy. We propose a medical-focused evaluation workflow that jointly assesses factual accuracy and coherence. Using GPT-Judger and further human validation, we introduce the Factuality-Weighted Score (FWS), a composite metric prioritizing factual accuracy beyond coherence to guide watermarking deployment in medical domains. Our evaluation shows current watermarking methods substantially compromise medical factuality, with entropy shifts degrading medical entity representation. These findings underscore the need for domain-aware watermarking approaches that preserve the integrity of medical content.

Problem

• Evaluating watermarking reliability in medical contexts
• Assessing factual accuracy beyond coherence tradeoffs
• Addressing medical factuality risks from entropy shifts

Innovation

• Medical-focused evaluation workflow assessing factual accuracy
• Factuality-Weighted Score metric prioritizing accuracy beyond coherence
• GPT-Judger with human validation for medical watermarking evaluation