X2-DFD: A framework for eXplainable and eXtendable Deepfake Detection

2024-10-08

arXiv.org

AI Summary

Existing deepfake detection methods often lack human-interpretable explanations; while multimodal large language models (MLLMs, e.g., LLaVA) have been explored, their capability to identify forensic traces and generate faithful, explanatory rationales remains limited. To address this, we propose X²-DFD—a novel framework introducing a tripartite synergistic paradigm: *feature assessment*, *strong-feature fine-tuning*, and *weak-feature external supplementation*. X²-DFD systematically evaluates MLLMs’ capacity to model forgery cues and explicitly enhances both discriminative power and interpretability. It integrates an MLLM backbone with automated forensic feature analysis, customized fine-tuning data construction, and a dedicated detector module. Extensive experiments demonstrate that X²-DFD achieves significant improvements in detection accuracy and explanation quality across multiple benchmarks, effectively balancing high discrimination performance with human-understandable reasoning.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Detecting deepfakes has become an important task. Most existing detection methods provide only real/fake predictions without offering human-comprehensible explanations. Recent studies leveraging MLLMs for deepfake detection have shown improvements in explainability. However, the performance of pre-trained MLLMs (e.g., LLaVA) remains limited due to a lack of understanding of their capabilities for this task and strategies to enhance them. In this work, we empirically assess the strengths and weaknesses of MLLMs specifically in deepfake detection via forgery features analysis. Building on these assessments, we propose a novel framework called ${X}^2$-DFD, consisting of three core modules. The first module, Model Feature Assessment (MFA), measures the detection capabilities of forgery features intrinsic to MLLMs, and gives a descending ranking of these features. The second module, Strong Feature Strengthening (SFS), enhances the detection and explanation capabilities by fine-tuning the MLLM on a dataset constructed based on the top-ranked features. The third module, Weak Feature Supplementing (WFS), improves the fine-tuned MLLM's capabilities on lower-ranked features by integrating external dedicated deepfake detectors. To verify the effectiveness of this framework, we further present a practical implementation, where an automated forgery features generation, evaluation, and ranking procedure is designed for MFA module; an automated generation procedure of the fine-tuning dataset containing real and fake images with explanations based on top-ranked features is developed for SFS model; an external conventional deepfake detector focusing on blending artifact, which corresponds to a low detection capability in the pre-trained MLLM, is integrated for WFS module. Experiments show that our approach enhances both detection and explanation performance.

Problem

• Existing deepfake detection lacks human-comprehensible explanations.
• Pre-trained MLLMs have limited performance in deepfake detection.
• Enhancing MLLMs' detection and explanation capabilities via feature analysis.

Innovation

• Empirically assesses MLLMs for deepfake detection
• Enhances MLLMs via feature ranking and fine-tuning
• Integrates external detectors for weak features