Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture

2024-08-23

arXiv.org

AI Summary

To address weak generalization and inefficient domain adaptation in open-set face forgery detection, this paper proposes a lightweight cross-domain adaptive method based on statistical modeling of forgery-style discrepancies. The approach tackles the problem by (1) introducing a forgery-style mixing augmentation mechanism that explicitly models inter-domain variations in forged textures and statistical distributions; and (2) designing a parameter-efficient adaptation framework tailored for Vision Transformers (ViTs), which fine-tunes only lightweight modules to jointly capture global and local forensic cues while fully preserving ImageNet pre-trained knowledge. Under the open-set setting, the method achieves state-of-the-art generalization performance, reduces trainable parameters by over 90%, and significantly enhances detection robustness and edge-deployment efficiency.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Application Domains: Misinformation & Fake News

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Open-set face forgery detection poses significant security threats and presents substantial challenges for existing detection models. These detectors primarily have two limitations: they cannot generalize across unknown forgery domains and inefficiently adapt to new data. To address these issues, we introduce an approach that is both general and parameter-efficient for face forgery detection. It builds on the assumption that different forgery source domains exhibit distinct style statistics. Previous methods typically require fully fine-tuning pre-trained networks, consuming substantial time and computational resources. In turn, we design a forgery-style mixture formulation that augments the diversity of forgery source domains, enhancing the model's generalizability across unseen domains. Drawing on recent advancements in vision transformers (ViT) for face forgery detection, we develop a parameter-efficient ViT-based detection model that includes lightweight forgery feature extraction modules and enables the model to extract global and local forgery clues simultaneously. We only optimize the inserted lightweight modules during training, maintaining the original ViT structure with its pre-trained ImageNet weights. This training strategy effectively preserves the informative pre-trained knowledge while flexibly adapting the model to the task of Deepfake detection. Extensive experimental results demonstrate that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters, representing an important step toward open-set Deepfake detection in the wild.

Problem

• Detecting unknown deepfake domains with limited generalization
• Reducing computational cost in adapting to new forgery data
• Enhancing model efficiency while preserving detection accuracy

Innovation

• Forgery style mixture formulation for domain diversity
• Parameter-efficient ViT model with lightweight modules
• Optimizing only inserted modules while preserving pre-trained weights