Forged Calamity: Benchmark for Cross-Domain Synthetic Disaster Detection in the Age of Diffusion

2026-06-16

AI Summary

This study addresses the emerging threat posed by AI-generated photorealistic disaster imagery, which can be mistaken for authentic content and thereby compromise cybersecurity, digital forensics, and emergency response efforts. The work presents the first benchmark focused on cross-domain synthetic disaster image detection, introducing a dataset of 30,000 images—comprising 6,000 real photographs and 24,000 generated by four state-of-the-art diffusion models—and systematically evaluates both fine-tuned classifiers and zero-shot general-purpose detectors. Experimental results demonstrate that fine-tuned methods perform well in-distribution but suffer accuracy drops of up to 50% under domain shift, while zero-shot approaches exhibit widespread instability, with only a few models showing limited generalization. The findings highlight a critical lack of robustness in current detection techniques when confronted with unseen generators or disaster types, underscoring the need for domain- and model-agnostic detection strategies.

Abstract

The rapid advancement of text-to-image diffusion models has enabled the creation of highly photorealistic synthetic images that closely resemble real photographs, making it increasingly difficult to distinguish authentic content from AI-generated fabrications. This poses challenges for cybersecurity, digital forensics, and disaster response, where fake imagery of floods, fires, or earthquakes can spread misinformation or disrupt emergency operations. To address this, we introduce Forged Calamity, a benchmark dataset for synthetic disaster detection containing 30,000 images, including 6,000 real and 24,000 synthetic samples generated by four diffusion models. Comprehensive experiments across fine-tuned and zero-shot settings reveal consistent weaknesses in current forensic approaches. Fine-tuned detectors perform well in-distribution but lose up to 50\% accuracy on unseen generators or disaster types, showing overfitting to model-specific artifacts. Zero-shot generalized detectors also struggle to maintain stable accuracy, with only limited resilience in a few representation-robust models. These findings highlight persistent generalization gaps and the urgent need for domain- and model-agnostic detection methods to ensure visual authenticity in the diffusion era.

Problem

• synthetic disaster detection
• diffusion models
• cross-domain
• image forgery
• visual authenticity

Innovation

• synthetic disaster detection
• diffusion models
• cross-domain generalization
• forgery benchmark
• zero-shot detection