Seeing, Hearing, and Knowing Together: Multimodal Strategies in Deepfake Videos Detection

2026-02-01

AI Summary

This study addresses the growing challenge of detecting increasingly realistic deepfake videos, where human perceptual capabilities are limited and media literacy interventions are urgently needed. Through an experiment involving 195 participants who judged real versus fake videos, provided confidence ratings, and reported their decision cues, we systematically uncover—via behavioral analysis, confidence calibration, and association rule mining—the multimodal interplay of visual, auditory, and intuitive signals in detection strategies. Our findings reveal that successful identification often hinges on combinations of appearance, voice, and gut feeling, with participants demonstrating higher accuracy and better-calibrated confidence for real videos. The study identifies specific cues that either facilitate or hinder detection, offering empirical foundations for targeted media literacy training.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

As deepfake videos become increasingly difficult for people to recognise, understanding the strategies humans use is key to designing effective media literacy interventions. We conducted a study with 195 participants between the ages of 21 and 40, who judged real and deepfake videos, rated their confidence, and reported the cues they relied on across visual, audio, and knowledge strategies. Participants were more accurate with real videos than with deepfakes and showed lower expected calibration error for real content. Through association rule mining, we identified cue combinations that shaped performance. Visual appearance, vocal, and intuition often co-occurred for successful identifications, which highlights the importance of multimodal approaches in human detection. Our findings show which cues help or hinder detection and suggest directions for designing media literacy tools that guide effective cue use. Building on these insights can help people improve their identification skills and become more resilient to deceptive digital media.

Problem

• deepfake detection
• multimodal strategies
• media literacy
• human perception
• audiovisual cues

Innovation

• multimodal strategies
• deepfake detection
• association rule mining
• media literacy
• human perception