A Lightweight and Interpretable Deepfakes Detection Framework

2025-01-21

AI Summary

To address insufficient physiological inconsistency modeling, poor multi-task generalization, and lack of interpretability in deepfake video detection—particularly for face swapping, lip-sync desynchronization, and puppeteering—this paper proposes a unified, lightweight joint detection framework. We introduce the first video-based heart rate signal estimation to model facial physiological inconsistency; design a hybrid representation integrating static facial landmarks, dynamic motion trajectories, and physiological heart rate features; and employ XGBoost to construct a high-accuracy, highly interpretable, and computationally efficient classifier. Evaluated on the WLDR benchmark, our method significantly outperforms state-of-the-art lightweight models, achieves performance comparable to deep architectures such as LSTM-FCN, and delivers real-time inference capability alongside transparent, human-understandable decision rationales.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The recent realistic creation and dissemination of so-called deepfakes poses a serious threat to social life, civil rest, and law. Celebrity defaming, election manipulation, and deepfakes as evidence in court of law are few potential consequences of deepfakes. The availability of open source trained models based on modern frameworks such as PyTorch or TensorFlow, video manipulations Apps such as FaceApp and REFACE, and economical computing infrastructure has easen the creation of deepfakes. Most of the existing detectors focus on detecting either face-swap, lip-sync, or puppet master deepfakes, but a unified framework to detect all three types of deepfakes is hardly explored. This paper presents a unified framework that exploits the power of proposed feature fusion of hybrid facial landmarks and our novel heart rate features for detection of all types of deepfakes. We propose novel heart rate features and fused them with the facial landmark features to better extract the facial artifacts of fake videos and natural variations available in the original videos. We used these features to train a light-weight XGBoost to classify between the deepfake and bonafide videos. We evaluated the performance of our framework on the world leaders dataset (WLDR) that contains all types of deepfakes. Experimental results illustrate that the proposed framework offers superior detection performance over the comparative deepfakes detection methods. Performance comparison of our framework against the LSTM-FCN, a candidate of deep learning model, shows that proposed model achieves similar results, however, it is more interpretable.

Problem

• Deepfake Detection
• Video Authenticity
• Biometric Forensics

Innovation

• Facial and Heartbeat Features
• XGBoost Model
• Deep Fake Detection