AI-GenBench: A New Ongoing Benchmark for AI-Generated Image Detection

2025-04-29

AI Summary

Rapid evolution of AI-generated image synthesis—from GANs to diffusion models—has undermined the reliability of generalization assessment for detection methods, as static benchmarks fail to reflect temporal shifts in generative capabilities. Method: This paper introduces the first time-evolving benchmark and time-aware evaluation framework. It dynamically partitions datasets according to the chronological development of generative models, integrates controllable data augmentation, standardized evaluation protocols, and a lightweight open-source toolchain to address static dataset bias, unfair cross-model comparisons, and excessive computational demands. Contribution/Results: We construct a high-quality, multi-source synthetic image dataset enabling robust cross-model generalization evaluation and non-expert-friendly deployment. Experiments demonstrate that our framework significantly improves the fidelity and reliability of detector generalization assessment on unseen generators. The complete codebase and dataset are publicly released, ensuring strong reproducibility, low entry barriers, and high extensibility.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Web Mining and Content Analysis: Models for Web evolution

Abstract

The rapid advancement of generative AI has revolutionized image creation, enabling high-quality synthesis from text prompts while raising critical challenges for media authenticity. We present Ai-GenBench, a novel benchmark designed to address the urgent need for robust detection of AI-generated images in real-world scenarios. Unlike existing solutions that evaluate models on static datasets, Ai-GenBench introduces a temporal evaluation framework where detection methods are incrementally trained on synthetic images, historically ordered by their generative models, to test their ability to generalize to new generative models, such as the transition from GANs to diffusion models. Our benchmark focuses on high-quality, diverse visual content and overcomes key limitations of current approaches, including arbitrary dataset splits, unfair comparisons, and excessive computational demands. Ai-GenBench provides a comprehensive dataset, a standardized evaluation protocol, and accessible tools for both researchers and non-experts (e.g., journalists, fact-checkers), ensuring reproducibility while maintaining practical training requirements. By establishing clear evaluation rules and controlled augmentation strategies, Ai-GenBench enables meaningful comparison of detection methods and scalable solutions. Code and data are publicly available to ensure reproducibility and to support the development of robust forensic detectors to keep pace with the rise of new synthetic generators.

Problem

• Detecting AI-generated images in real-world scenarios
• Generalizing detection methods to new generative models
• Overcoming limitations in current detection approaches

Innovation

• Temporal evaluation framework for AI detection
• Standardized protocol for fair model comparison
• Public dataset and tools for reproducibility