🤖 AI Summary
This study addresses the severe threat posed by highly realistic fake news videos generated by generative AI to the authenticity of digital media. To counter this challenge, the authors propose a multimodal deepfake detection method that innovatively integrates lip motion, speech, and facial visual features. Temporal and semantic information from these modalities are extracted using LipNet, DeepSpeech2, and ResNet18—augmented with BlazeFace for face detection—respectively. A decision-level fusion is then performed via an ensemble classifier combining Random Forest, Multilayer Perceptron (MLP), and Long Short-Term Memory (LSTM) networks. Evaluated on the FakeAVCeleb dataset, the proposed approach achieves a detection accuracy of 94%, significantly outperforming existing baselines and demonstrating enhanced robustness and generalization capability.
📝 Abstract
Deepfake news refers to AI-generated (or AI ma-nipulated) multimedia content intentionally generated to deceive audiences by manipulating the facial expressions, or speech while maintaining the realistic appearance. The rapid progress of generative AI has made the synthesis of highly realistic fake videos and cloned voices widely accessible, posing a serious threat to the authenticity of digital news media. This paper presents a multi-modal framework that discerns the authenticity of video content by jointly exploiting audio and visual cues, thereby addressing the challenge of detecting the deepfake videos. We proposed a framework that involves features extraction from lip movements, audio content and video frames. Lip movements and speech content are encoded using the LipNet and DeepSpeech2 models, while facial features are extracted by leveraging the use of BlazeFace and represented with ResNet18. The extracted feature vectors are concatenated into a holistic video representation and classified with an ensemble of machine learning and deep learning models, including Random Forest (RF), Multi-layer Perceptron (MLP) and Long Short-Term Memory (LSTM) networks. Exten-sive experiments performed on the FakeAVCeleb dataset shows that the proposed approach attains an accuracy of 94% using augmented audio features, outperforming a state-of-the-art multi-modal ensemble baseline. The results confirm the robustness and practical potential of the proposed framework for deepfake news detection.