Problem
Research questions and friction points this paper is trying to address.
video face forgery detection
partial forgery
face manipulation
deepfakes
multimodal representation
Learning Unified Video and Image Representation for Video Face Forgery Detection
🤖 AI Summary
Existing methods struggle to effectively detect deepfake videos where only a subset of frames has been manipulated. To address this challenge, this work proposes the UVIF framework, which introduces annotated static images to provide fine-grained supervision for video frames. UVIF employs a unified 2D backbone to jointly model both images and videos, aligning cross-modal representations through multi-task learning, a pseudo-labeling mechanism, and a temporal fusion module. Notably, this approach achieves significant performance gains over current state-of-the-art methods across multiple benchmark datasets without incurring additional computational overhead, thereby substantially improving detection accuracy for partially forged videos.
📝 Abstract
Face forgery detection is crucial for preserving the security and integrity of facial data given the rapid developments in face manipulation techniques and deep generative models. Existing methods for video face forgery detection typically assume that all frames in a forged video are manipulated, while detecting partially forged videos that contain only a subset of altered frames remains challenging. To address this issue, we propose a novel framework, UVIF, that utilizes additional annotated images to provide fine-grained supervision for detecting partial forgeries in videos. UVIF employs a unified encoder and a multi-task learning paradigm to jointly model facial videos and images for boosted video face forgery detection. A 2D backbone with temporal fusion modules is employed as the unified encoder. A pseudo labeling process is designed for video frames to bridge their representations with those of static images. A video-oriented feature alignment strategy is further introduced to reduce the distribution gap between videos and images. Extensive experiments on benchmark datasets demonstrate the effectiveness of our framework, which outperforms state-of-theart methods in detecting partially forged videos while introducing no additional computational overhead. Our code is available at https://github.com/haotianll/UVIF.
Innovation
Methods, ideas, or system contributions that make the work stand out.
unified representation
partial forgery detection
multi-task learning
feature alignment
pseudo labeling
🔎 Similar Papers
No similar papers found.
