LLM Output Detectability and Task Performance Can be Jointly Optimized

2026-05-02

AI Summary

Existing watermarking methods for large language models often suffer from degraded task performance, limiting their practicality. This work proposes PUPPET, a novel framework that, for the first time, jointly optimizes output detectability and downstream task performance through reinforcement learning. By leveraging a dual-reward signal from both a watermark detector and a task evaluator, PUPPET enables efficient fine-tuning with only a few thousand samples and 1–2 GPU hours. The approach overcomes the performance degradation bottleneck of conventional watermarking techniques, significantly outperforming existing methods across diverse tasks such as long-form question answering, summarization, and argumentative essay generation. Moreover, PUPPET demonstrates strong robustness—maintaining high detection rates while exhibiting cross-model transferability, domain generalization, and resilience against paraphrasing attacks.

Abstract

Detecting machine-generated text is essential for transparency and accountability when deploying large language models (LLMs). Among detection approaches, watermarking is a statistically reliable method by design -- it embeds detectable signals into LLM outputs by biasing their token distributions. However, it has been reported that watermarked LLMs often perform worse on downstream tasks. We propose PUPPET, a framework that fine-tunes an LLM via reinforcement learning to generate text that is both more detectable and better performing on downstream tasks. We use two reward functions: a detector that outputs a machine-class likelihood and an evaluator that measures a task-specific metric. Experiments on long-form QA, summarization, and essay writing show that LLMs trained with PUPPET achieve high detectability competitive with watermarking methods while outperforming them on downstream tasks. The analysis shows that this optimization can be performed efficiently with only a few thousand samples in 1--2 GPU hours. Moreover, these gains are consistent across out-of-domain tasks, different LLM families, and model sizes, and are even robust to paraphrasing attacks.

Problem

• LLM output detectability
• watermarking
• task performance
• machine-generated text detection
• downstream task degradation

Innovation

• watermarking
• reinforcement learning
• detectability
• task performance
• robustness