Audio-visual Deepfake Detection With Local Temporal Inconsistencies

2025-01-14

AI Summary

This paper addresses the challenge of detecting fine-grained temporal asynchrony—such as subtle lip-voice misalignment—in audiovisual deepfakes. We propose a novel method that jointly models temporal discrepancies via a time-distance map and cross-modal attention. Key contributions include: (1) the first local fake data synthesis technique explicitly designed to emulate realistic asynchronous forgery patterns; and (2) a temporal distance mapping module integrated with a multimodal feature alignment-contrastive learning framework, which explicitly captures local cross-modal temporal inconsistencies. Evaluated on DFDC and FakeAVCeleb benchmarks, our approach significantly outperforms state-of-the-art methods—particularly in detecting lip-voice desynchronization with sub-100ms latency—demonstrating superior discriminative capability for fine-grained asynchronous anomalies.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance

Abstract

This paper proposes an audio-visual deepfake detection approach that aims to capture fine-grained temporal inconsistencies between audio and visual modalities. To achieve this, both architectural and data synthesis strategies are introduced. From an architectural perspective, a temporal distance map, coupled with an attention mechanism, is designed to capture these inconsistencies while minimizing the impact of irrelevant temporal subsequences. Moreover, we explore novel pseudo-fake generation techniques to synthesize local inconsistencies. Our approach is evaluated against state-of-the-art methods using the DFDC and FakeAVCeleb datasets, demonstrating its effectiveness in detecting audio-visual deepfakes.

Problem

• Audio-Visual Synchronization
• Content Authenticity
• Multimedia Forensics

Innovation

• Audio-Visual Synchronization
• Attention Mechanism
• False Data Generation