Is Artificial Intelligence Generated Image Detection a Solved Problem?

2025-05-18

AI Summary

Has AI-generated image (AIGI) detection been truly solved? Despite strong performance of existing detectors on standard benchmarks, their robustness and generalization in real-world scenarios remain highly questionable. This paper introduces AIGIBench—the first comprehensive benchmark addressing four critical real-world challenges: multi-source generalization, robustness to image degradation, sensitivity to data augmentation, and impact of test-time preprocessing. Built upon a high-quality evaluation set comprising images from 23 generative models (including GANs and diffusion models) and authentic social-media sources, AIGIBench systematically evaluates 11 state-of-the-art detectors. Results reveal an average performance drop exceeding 40%, widespread failure of mainstream augmentations, and highly nonlinear effects of preprocessing. These findings expose a substantial performance gap between controlled experiments and practical deployment, establishing a reproducible, high-fidelity evaluation paradigm for AIGI detection research.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Philosophy and Ethics of AI: Artificial General Intelligence
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

The rapid advancement of generative models, such as GANs and Diffusion models, has enabled the creation of highly realistic synthetic images, raising serious concerns about misinformation, deepfakes, and copyright infringement. Although numerous Artificial Intelligence Generated Image (AIGI) detectors have been proposed, often reporting high accuracy, their effectiveness in real-world scenarios remains questionable. To bridge this gap, we introduce AIGIBench, a comprehensive benchmark designed to rigorously evaluate the robustness and generalization capabilities of state-of-the-art AIGI detectors. AIGIBench simulates real-world challenges through four core tasks: multi-source generalization, robustness to image degradation, sensitivity to data augmentation, and impact of test-time pre-processing. It includes 23 diverse fake image subsets that span both advanced and widely adopted image generation techniques, along with real-world samples collected from social media and AI art platforms. Extensive experiments on 11 advanced detectors demonstrate that, despite their high reported accuracy in controlled settings, these detectors suffer significant performance drops on real-world data, limited benefits from common augmentations, and nuanced effects of pre-processing, highlighting the need for more robust detection strategies. By providing a unified and realistic evaluation framework, AIGIBench offers valuable insights to guide future research toward dependable and generalizable AIGI detection.

Problem

• Evaluating robustness of AI-generated image detectors in real-world scenarios
• Assessing generalization of detectors across diverse synthetic image sources
• Analyzing impact of image degradation and processing on detection accuracy

Innovation

• Introduces AIGIBench for robust AIGI detection evaluation
• Simulates real-world challenges with four core tasks
• Tests 11 detectors on 23 diverse fake image subsets