SELFI: Selective Fusion of Identity for Generalizable Deepfake Detection

2025-06-21

AI Summary

In deepfake detection, suboptimal utilization of identity features—either through blind suppression or excessive reliance—severely limits cross-method generalization. To address this, we propose a sample-adaptive dynamic identity feature regulation framework. Our method explicitly models identity features and selectively fuses them in a demand-driven manner, introducing two key components: a Forgery-Aware Identity Adapter and a Correlation-Driven Identity-Visual Fusion Module. Identity embeddings are extracted from a frozen face recognition model; we further employ forgery-aware supervised fine-tuning, a gating mechanism, and attention-guided multimodal fusion. The resulting architecture is end-to-end trainable and lightweight. Evaluated on four benchmarks, our approach achieves an average AUC improvement of 3.1% and outperforms the state-of-the-art by 6% on DFDC, demonstrating significantly enhanced generalization to unseen deepfake generation methods.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Face identity provides a powerful signal for deepfake detection. Prior studies show that even when not explicitly modeled, classifiers often learn identity features implicitly. This has led to conflicting views: some suppress identity cues to reduce bias, while others rely on them as forensic evidence. To reconcile these views, we analyze two hypotheses: (1) whether face identity alone is discriminative for detecting deepfakes, and (2) whether such identity features generalize poorly across manipulation methods. Our experiments confirm that identity is informative but context-dependent. While some manipulations preserve identity-consistent artifacts, others distort identity cues and harm generalization. We argue that identity features should neither be blindly suppressed nor relied upon, but instead be explicitly modeled and adaptively controlled based on per-sample relevance. We propose extbf{SELFI} ( extbf{SEL}ective extbf{F}usion of extbf{I}dentity), a generalizable detection framework that dynamically modulates identity usage. SELFI consists of: (1) a Forgery-Aware Identity Adapter (FAIA) that extracts identity embeddings from a frozen face recognition model and projects them into a forgery-relevant space via auxiliary supervision; and (2) an Identity-Aware Fusion Module (IAFM) that selectively integrates identity and visual features using a relevance-guided fusion mechanism. Experiments on four benchmarks show that SELFI improves cross-manipulation generalization, outperforming prior methods by an average of 3.1% AUC. On the challenging DFDC dataset, SELFI exceeds the previous best by 6%. Code will be released upon paper acceptance.

Problem

• Analyzes if face identity alone detects deepfakes effectively
• Examines identity feature generalization across manipulation methods
• Proposes adaptive identity usage for generalizable deepfake detection

Innovation

• Selective fusion of identity features adaptively
• Forgery-aware identity adapter with auxiliary supervision
• Relevance-guided fusion of identity and visual features