Problem
Research questions and friction points this paper is trying to address.
Deepfake detectors
smartphone photography
neural image-signal processing
false-alarm rates
dataset
LAION-Mobile: Evaluating Deepfake Detectors On One Million Smartphone Photos
🤖 AI Summary
研究通过构建包含100万张智能手机照片的数据集LAION-Mobile，评估了12种深度伪造检测器在现代手机摄影上的表现，发现这些检测器对现代AI内容的识别效果不佳。
📝 Abstract
Most Deepfake detectors report near-perfect AUC scores on their reference benchmarks. However, a recent ICML position paper argues that these evaluations collectively neglect the impact of modern smartphone photography: the widely used on-device neural image-signal processing pipelines (like multi-sensor fusion or noise and motion-blur suppression) increasingly shift the imaging paradigm from simple lens projections towards computational photography. Hence, devices actually generate, rather than record photos. This increases the risk that deepfake detectors may flag ordinary phone photos as fake. Due to the lack of large-scale datasets containing images from modern smartphones, this hypothesis has so far only been tested in small proof-of-concept studies. The aim of this paper is to close this gap. We introduce LAION-Mobile, an open dataset containing about 1 million smartphone images with EXIF metadata distilled from re-LAION-5B. Evaluating twelve state-of-the-art deepfake detectors with their original paper checkpoints on a 9,115-image evaluation sample of this pool (DIRE on 738), we report three key findings: (i) On modern AI content no detector exceeds AUC 0.624, and five of twelve fall below chance. (ii) Real-photo false-alarm rates are an artefact of threshold calibration: thresholds fitted on legacy GAN data make several detectors look deployable (less than 11 percent FPR), yet the same detectors flag 17-91 percent of real photos once the identical criterion is refit on modern content. (iii) Consequently, no detector both beats chance on modern AI content and keeps a deployable real-photo false-alarm rate. Mirroring the device mix of web collections, the corpus probes the first neural-ISP generation (2018-2020); current flagships are essentially absent, leaving the modern-ISP regime as the open gap.
Innovation
Methods, ideas, or system contributions that make the work stand out.
LAION-Mobile
deepfake detectors
smartphone photography
neural image-signal processing (ISP)
false-alarm rates
🔎 Similar Papers
No similar papers found.
