AI-generated Text Detection: A Multifaceted Approach to Binary and Multiclass Classification

2025-05-15

AI Summary

To address the misuse of large language model (LLM)-generated text in disinformation, spam, and academic misconduct, this work proposes a dual-task detection framework: human-vs-machine binary classification (Task A) and multi-source LLM attribution (Task B). Methodologically, we design a lightweight, interpretable dual-neural architecture integrating a fine-tuned Transformer encoder, robust textual representation learning, feature-adaptive fusion, and task-specific classification heads. To our knowledge, this is the first approach to jointly optimize both tasks on the AAAI 2025 Defactify shared task: it achieves an F1 score of 0.994 on Task A (ranked 5th globally) and 0.627 on Task B (also 5th globally), substantially outperforming baselines. Our core contribution lies in unifying human-machine discrimination and LLM provenance attribution within a single, accurate, lightweight, and interpretable framework.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in generating text that closely resembles human writing across a wide range of styles and genres. However, such capabilities are prone to potential misuse, such as fake news generation, spam email creation, and misuse in academic assignments. As a result, accurate detection of AI-generated text and identification of the model that generated it are crucial for maintaining the responsible use of LLMs. In this work, we addressed two sub-tasks put forward by the Defactify workshop under AI-Generated Text Detection shared task at the Association for the Advancement of Artificial Intelligence (AAAI 2025): Task A involved distinguishing between human-authored or AI-generated text, while Task B focused on attributing text to its originating language model. For each task, we proposed two neural architectures: an optimized model and a simpler variant. For Task A, the optimized neural architecture achieved fifth place with $F1$ score of 0.994, and for Task B, the simpler neural architecture also ranked fifth place with $F1$ score of 0.627.

Problem

• Detecting AI-generated text to prevent misuse in fake news and spam.
• Identifying the specific language model that produced AI-generated text.
• Developing neural architectures for binary and multiclass classification tasks.

Innovation

• Binary and multiclass classification for AI text detection
• Optimized and simpler neural architectures proposed
• Achieved high F1 scores in both detection tasks