Forensics Adapter: Adapting CLIP for Generalizable Face Forgery Detection

2024-11-29

arXiv.org

AI Summary

Existing CLIP-based face forgery detection methods treat CLIP solely as a static feature extractor, lacking task-specific adaptation capability and suffering from limited generalization. To address this, we propose Forensics Adapter—a lightweight, forgery-boundary-aware adapter (5.7M parameters) that guides CLIP to explicitly model and fuse forensic traces. We further introduce a cross-module visual token interaction mechanism to enhance inter-layer propagation of localized forgery cues. Our framework marks the first paradigm shift transforming CLIP from a generic feature extractor into a task-driven, adaptable forgery detector. Evaluated on five standard benchmarks, our method achieves an average accuracy improvement of approximately 7% over state-of-the-art CLIP-based baselines, establishing a new benchmark for CLIP-based face forgery detection.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

We describe the Forensics Adapter, an adapter network designed to transform CLIP into an effective and generalizable face forgery detector. Although CLIP is highly versatile, adapting it for face forgery detection is non-trivial as forgery-related knowledge is entangled with a wide range of unrelated knowledge. Existing methods treat CLIP merely as a feature extractor, lacking task-specific adaptation, which limits their effectiveness. To address this, we introduce an adapter to learn face forgery traces -- the blending boundaries unique to forged faces, guided by task-specific objectives. Then we enhance the CLIP visual tokens with a dedicated interaction strategy that communicates knowledge across CLIP and the adapter. Since the adapter is alongside CLIP, its versatility is highly retained, naturally ensuring strong generalizability in face forgery detection. With only 5.7M trainable parameters, our method achieves a significant performance boost, improving by approximately 7% on average across five standard datasets. We believe the proposed method can serve as a baseline for future CLIP-based face forgery detection methods. The code is available at https://github.com/OUC-VAS/ForensicsAdapter.

Problem

• Adapt CLIP for generalizable face forgery detection
• Disentangle forgery knowledge from unrelated CLIP features
• Enhance detection with adapter learning blending boundaries

Innovation

• Adapter network enhances CLIP for forgery detection
• Task-specific objectives guide forgery trace learning
• Knowledge interaction strategy boosts CLIP tokens