When AUC Misleads: Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift

2026-06-17

AI Summary

This work addresses the limitation of existing deepfake detectors, whose cross-domain generalization performance is inadequately assessed by dataset-specific AUC scores that fail to reflect real-world scenarios involving mixed data distributions. To overcome this, the authors propose Cross-AUC, a unified and interpretable evaluation framework that introduces the concept of prediction polarization and leverages Wasserstein distance to quantify the divergence between the score distributions of real and fake samples. By integrating AUC metrics across multiple domains, Cross-AUC provides a more realistic assessment of detector robustness under domain shift. Experiments on seven benchmark datasets demonstrate that Cross-AUC more accurately reveals the underlying causes of performance degradation in cross-domain settings, substantially enhancing the practical relevance and utility of deepfake detection evaluation.

Abstract

Recent advances in generative AI, such as diffusion models and face-swapping tools, have enabled the creation of highly realistic deepfakes, leading to real-world harms including financial fraud and non-consensual explicit content. In response, deepfake detection has become an active research area, with recent methods increasingly focusing on improving generalization to unseen manipulations. This is typically evaluated using the Area Under the ROC Curve (AUC) measured separately across multiple datasets. However, such an evaluation fails to reflect real-world scenarios where detectors face a mixture of data sources and varying artifact types. To address this limitation, we introduce a novel metric, Cross-dataset AUC (Cross-AUC) that averages per-domain AUCs with a measure of prediction polarization for taking into account the robustness to domain shift. The polarization extent is quantified by the Wasserstein Distance between class score distributions. Cross-AUC not only assesses the generalization capabilities of deepfake detectors under domain shifts more realistically, but it is also interpretable as it better explains the reason behind a drop in performance. Experiments performed on seven benchmark datasets demonstrate its practical relevance.

Problem

• deepfake detection
• domain shift
• AUC
• evaluation metric
• generalization

Innovation

• Cross-AUC
• domain shift
• deepfake detection
• polarization-aware evaluation
• Wasserstein Distance