Explaining Generalization of AI-Generated Text Detectors Through Linguistic Analysis

2026-01-12

AI Summary

This study addresses the limited generalization of current AI-generated text detectors under cross-prompt, cross-model, and cross-domain settings, where failure mechanisms remain poorly understood. The authors construct a comprehensive benchmark encompassing six prompting strategies, seven large language models, and four domain-specific datasets to systematically evaluate detector robustness. For the first time, they establish a quantitative link between distributional shifts in an 80-dimensional linguistic feature space and detection accuracy. Their analysis reveals that variations in key linguistic attributes—such as tense usage and pronoun frequency—significantly degrade generalization performance. These findings provide both interpretable theoretical insights and empirical evidence to guide the development of more robust detection systems.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

AI-text detectors achieve high accuracy on in-domain benchmarks, but often struggle to generalize across different generation conditions such as unseen prompts, model families, or domains. While prior work has reported these generalization gaps, there are limited insights about the underlying causes. In this work, we present a systematic study aimed at explaining generalization behavior through linguistic analysis. We construct a comprehensive benchmark that spans 6 prompting strategies, 7 large language models (LLMs), and 4 domain datasets, resulting in a diverse set of human- and AI-generated texts. Using this dataset, we fine-tune classification-based detectors on various generation settings and evaluate their cross-prompt, cross-model, and cross-dataset generalization. To explain the performance variance, we compute correlations between generalization accuracies and feature shifts of 80 linguistic features between training and test conditions. Our analysis reveals that generalization performance for specific detectors and evaluation conditions is significantly associated with linguistic features such as tense usage and pronoun frequency.

Problem

• generalization
• AI-generated text detection
• linguistic analysis
• feature shift
• large language models

Innovation

• linguistic analysis
• generalization
• AI-generated text detection
• feature shift
• large language models