Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors

2025-05-21

AI Summary

This work addresses the vulnerability of large language model (LLM)-generated text to detection by automated classifiers. We propose CoPA, a training-free contrastive paraphrasing attack that leverages black-box LLMs via instruction engineering and implicit word-distribution modeling. CoPA is the first method to explicitly model the machine-generated word distribution as a subtractable auxiliary signal and integrate it into a contrastive decoding mechanism during generation to suppress detectable “machine artifacts.” Evaluated against state-of-the-art detectors—including DetectGPT and Fast-DetectGPT—across diverse text domains, CoPA achieves significant improvements in attack success rate while incurring zero training overhead. Its core innovations are: (i) a training-free contrastive decoding paradigm that dynamically steers token selection away from detector-sensitive patterns, and (ii) an operationalizable, explicit modeling of machine-induced lexical distributions as manipulable signals.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The misuse of large language models (LLMs), such as academic plagiarism, has driven the development of detectors to identify LLM-generated texts. To bypass these detectors, paraphrase attacks have emerged to purposely rewrite these texts to evade detection. Despite the success, existing methods require substantial data and computational budgets to train a specialized paraphraser, and their attack efficacy greatly reduces when faced with advanced detection algorithms. To address this, we propose extbf{Co}ntrastive extbf{P}araphrase extbf{A}ttack (CoPA), a training-free method that effectively deceives text detectors using off-the-shelf LLMs. The first step is to carefully craft instructions that encourage LLMs to produce more human-like texts. Nonetheless, we observe that the inherent statistical biases of LLMs can still result in some generated texts carrying certain machine-like attributes that can be captured by detectors. To overcome this, CoPA constructs an auxiliary machine-like word distribution as a contrast to the human-like distribution generated by the LLM. By subtracting the machine-like patterns from the human-like distribution during the decoding process, CoPA is able to produce sentences that are less discernible by text detectors. Our theoretical analysis suggests the superiority of the proposed attack. Extensive experiments validate the effectiveness of CoPA in fooling text detectors across various scenarios.

Problem

• Detecting LLM-generated texts to prevent misuse like plagiarism
• Existing paraphrase attacks require heavy training resources
• Overcoming inherent machine-like biases in LLM outputs for evasion

Innovation

• Training-free method using off-the-shelf LLMs
• Contrastive human-like and machine-like distributions
• Instruction crafting for human-like text generation