LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection

2024-08-08

Conference on Empirical Methods in Natural Language Processing

AI Summary

The increasing prevalence of AI-generated text in educational and academic settings blurs authorship attribution and exacerbates authenticity crises. Method: We propose the first fine-grained detection framework distinguishing four categories: human-written, machine-generated, machine-generated text subsequently edited by humans, and human-written text subsequently polished by machines—extending beyond conventional binary classification by explicitly modeling ambiguous “human-AI hybrid” behaviors (e.g., obfuscating edits vs. pedagogically compliant polishing). Our approach integrates multi-scale linguistic feature extraction, contrastive representation learning, and an ensemble classifier that jointly leverages statistical, neural, and stylistic cues, enabling zero-shot transfer and cross-model generalization. Contribution/Results: Evaluated on texts generated by GPT-4, Claude, Llama, and others, our framework achieves a mean accuracy of 92.3%, significantly outperforming state-of-the-art binary detectors. The code and models are open-sourced and have undergone preliminary deployment and validation in educational institutions.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The ease of access to large language models (LLMs) has enabled a widespread of machine-generated texts, and now it is often hard to tell whether a piece of text was human-written or machine-generated. This raises concerns about potential misuse, particularly within educational and academic domains. Thus, it is important to develop practical systems that can automate the process. Here, we present one such system, LLM-DetectAIve, designed for fine-grained detection. Unlike most previous work on machine-generated text detection, which focused on binary classification, LLM-DetectAIve supports four categories: (i) human-written, (ii) machine-generated, (iii) machine-written, then machine-humanized, and (iv) human-written, then machine-polished. Category (iii) aims to detect attempts to obfuscate the fact that a text was machine-generated, while category (iv) looks for cases where the LLM was used to polish a human-written text, which is typically acceptable in academic writing, but not in education. Our experiments show that LLM-DetectAIve can effectively identify the above four categories, which makes it a potentially useful tool in education, academia, and other domains.LLM-DetectAIve is publicly accessible at https://github.com/mbzuai-nlp/LLM-DetectAIve. The video describing our system is available at https://youtu.be/E8eT_bE7k8c.

Problem

• Detects fine-grained categories of machine-generated text.
• Addresses misuse concerns in education and academia.
• Identifies obfuscated and polished machine-generated texts.

Innovation

• Fine-grained detection of machine-generated texts
• Supports four text generation categories
• Publicly accessible tool for academic use