Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift

2026-06-23

AI Summary

This study addresses the sharp performance degradation of existing AI-generated text detectors in real-world deployment due to persistent distribution shifts—such as adversarial rewrites, the emergence of new language models, and evolving human writing styles—compounded by a lack of labeled data. To tackle this challenge, the work introduces test-time adaptation (TTA) to the domain for the first time, proposing a semi-supervised adaptation method that leverages the homogeneity of unlabeled samples during inference to dynamically recalibrate the detector without requiring labels. Experimental results demonstrate that the proposed approach achieves 90.5% accuracy on adversarial AI-text detection, substantially outperforming the commercial system Pangram (24.1%). This advancement effectively overcomes the limitations of static supervised models and significantly enhances the robustness and reliability of detection systems in dynamic, real-world environments.

Abstract

Deployed approaches for AI text detection often rely on training-time access to labeled datasets of both human-written and AI-generated text. This approach is vulnerable to three types of distribution shifts that occur continually post-deployment, and for which labeled data is often unavailable: adversarial humanization, new LLMs being released, and temporal drift in human writing. Simultaneously, existing approaches do not leverage a key signal of LLM usage: inference-time homogeneity. We propose a test-time adaptation (TTA) approach, using semi-supervised learning, that adapts to distribution shifts by leveraging homogeneity among unlabeled samples observed at inference time. Empirically, we find that state-of-the-art supervised detectors systematically fail when they encounter distribution shifts in AI-generated and human writing, both adversarial and natural, while test-time adaptation with semi-supervised learning is largely robust; e.g., the commercial model Pangram detects just 24.1% of our adversarial AI-generated text, compared to 90.5% for our test-time approach. We establish that test-time adaptation is a promising framework for AI text detection in the wild. We publicly release our code (which includes code for model training, evaluation, and plots) at https://github.com/kkr36/llm_detection.

Problem

• distribution shift
• AI text detection
• test-time adaptation
• adversarial humanization
• temporal drift

Innovation

• test-time adaptation
• AI text detection
• distribution shift
• semi-supervised learning
• inference-time homogeneity