Online Detecting LLM-Generated Texts via Sequential Hypothesis Testing by Betting

2024-10-29

arXiv.org

AI Summary

To address the challenge of real-time detection of LLM-generated text in streaming-content scenarios, this paper proposes an online detection framework based on betting-based sequential testing. Our method pioneers the integration of sequential hypothesis testing with gambling strategies, establishing a unified framework for probabilistic calibration and streaming statistical decision-making, and provides provable upper bounds on detection delay while strictly controlling the false positive rate (FPR). Compared to existing offline methods and heuristic online baselines, our approach achieves 42–68% average latency reduction on text generated by mainstream LLMs—including GPT-4, Claude, and Llama—while maintaining FPR < 1%. Extensive experiments demonstrate its effectiveness and robustness under high-throughput, low-latency conditions typical of news websites and social media platforms. To the best of our knowledge, this is the first theoretically grounded solution for LLM content governance that simultaneously ensures statistical reliability and real-time performance.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Developing algorithms to differentiate between machine-generated texts and human-written texts has garnered substantial attention in recent years. Existing methods in this direction typically concern an offline setting where a dataset containing a mix of real and machine-generated texts is given upfront, and the task is to determine whether each sample in the dataset is from a large language model (LLM) or a human. However, in many practical scenarios, sources such as news websites, social media accounts, or on other forums publish content in a streaming fashion. Therefore, in this online scenario, how to quickly and accurately determine whether the source is an LLM with strong statistical guarantees is crucial for these media or platforms to function effectively and prevent the spread of misinformation and other potential misuse of LLMs. To tackle the problem of online detection, we develop an algorithm based on the techniques of sequential hypothesis testing by betting that not only builds upon and complements existing offline detection techniques but also enjoys statistical guarantees, which include a controlled false positive rate and the expected time to correctly identify a source as an LLM. Experiments were conducted to demonstrate the effectiveness of our method.

Problem

• Detect machine-generated texts in online streaming scenarios
• Ensure quick and accurate identification of LLM sources
• Control false positives and guarantee statistical reliability

Innovation

• Sequential hypothesis testing for online detection
• Controlled false positive rate guarantees
• Fast identification of LLM-generated texts