Are DeepFakes Realistic Enough? Exploring Semantic Mismatch as a Novel Challenge

2026-04-30

AI Summary

This work addresses the limitations of existing DeepFake detection methods, which are predominantly confined to binary classification and fail to handle audio-only, video-only, or joint audiovisual manipulations while overlooking the critical challenge of semantic inconsistency. To bridge this gap, the study proposes the first four-class evaluation framework, introducing a novel category—“real audio and real video but semantically mismatched” (RARV-SMM)—and establishes a multimodal detection benchmark using the FakeAVCeleb and LAV-DF datasets. By designing three audiovisual discrepancy variants, the authors demonstrate a significant performance drop in current state-of-the-art models under semantic inconsistency. They further introduce a semantics-aware training strategy leveraging ImageBind multimodal embeddings, which consistently improves detection accuracy across both established and newly proposed evaluation settings, advancing DeepFake detection toward more realistic scenarios.

Abstract

Current DeepFake detection scenarios are mostly binary, yet data manipulation can vary across audio, video, or both, whose variability is not captured in binary settings. Four-class audio-visual formulations address this by discriminating manipulation type, but introduce a unresolved problem: models may rely solely on data source integrity to detect DeepFakes without evaluating their semantic consistency. If the DeepFake origin is not in the data source but in its content, can semantic mismatch be assessed by the state-of-the-art? This paper proposes a new evaluation setup, extending the four-class formulation by explicitly modeling semantic-level inconsistency between authentic modalities with the introduction a new class: Real Audio-Real Video with Semantic Mismatch (RARV-SMM). We assess the robustness of state-of-the-art models in this new realistic DeepFake setting, using the FakeAVCeleb dataset, highlighting the limitations of existing approaches when faced with semantic mismatch data. We further introduce three RARV-SMM variants that expose distinct architectural vulnerabilities as audio-visual divergence increases. We also propose a semantic reinforcement strategy that incorporates the semantic mismatch class and ImageBind embeddings to improve DeepFake detection in both our proposed and state-of-the-art settings, on FakeAVCeleb and LAV-DF, paving the way to more realistic DeepFake detectors. The source code and data are available at https://github.com/.

Problem

• DeepFake detection
• semantic mismatch
• audio-visual inconsistency
• manipulation type
• realistic DeepFakes

Innovation

• Semantic Mismatch
• DeepFake Detection
• Audio-Visual Inconsistency
• ImageBind Embeddings
• RARV-SMM