Unraveling Hidden Representations: A Multi-Modal Layer Analysis for Better Synthetic Content Forensics

2025-08-01

AI Summary

Existing deepfake detection methods suffer from poor generalization, particularly across modalities (e.g., images vs. audio) and diverse generative models. This work proposes a universal forgery detection framework grounded in the latent representations of pre-trained multimodal foundation models. We empirically discover an inherent separability between authentic and synthetic samples within the frozen feature space—enabling effective detection via only a lightweight linear classifier trained atop fixed encoders. The approach unifies image and audio deepfake detection under a single paradigm, achieving few-shot adaptation, rapid deployment, and strong cross-modal and cross-architectural generalization. Evaluated on multiple benchmarks, it surpasses state-of-the-art methods in detection accuracy, robustness, and computational efficiency. Crucially, it demonstrates superior zero-shot or few-shot transfer to unseen generative models—significantly enhancing plug-and-play detection capability against emerging synthesis technologies.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Generative models achieve remarkable results in multiple data domains, including images and texts, among other examples. Unfortunately, malicious users exploit synthetic media for spreading misinformation and disseminating deepfakes. Consequently, the need for robust and stable fake detectors is pressing, especially when new generative models appear everyday. While the majority of existing work train classifiers that discriminate between real and fake information, such tools typically generalize only within the same family of generators and data modalities, yielding poor results on other generative classes and data domains. Towards a universal classifier, we propose the use of large pre-trained multi-modal models for the detection of generative content. Effectively, we show that the latent code of these models naturally captures information discriminating real from fake. Building on this observation, we demonstrate that linear classifiers trained on these features can achieve state-of-the-art results across various modalities, while remaining computationally efficient, fast to train, and effective even in few-shot settings. Our work primarily focuses on fake detection in audio and images, achieving performance that surpasses or matches that of strong baseline methods.

Problem

• Detecting synthetic content across multiple data modalities
• Improving generalization of fake detectors beyond specific generators
• Achieving efficient and accurate fake detection in audio and images

Innovation

• Uses large pre-trained multi-modal models
• Analyzes latent code for fake detection
• Trains efficient linear classifiers across modalities