Anomaly Detection and Localization for Speech Deepfakes via Feature Pyramid Matching

2025-03-23

AI Summary

To address poor generalization and lack of interpretability in deepfake speech detection, this paper proposes an interpretable one-class anomaly detection framework trained exclusively on genuine speech. We reformulate deepfake detection as an out-of-distribution (OOD) anomaly detection task—the first such formulation in the domain. Our method introduces a Student-Teacher feature pyramid matching mechanism and a novel discrepancy scaling regularization strategy, enabling strong cross-generator and cross-lingual generalization alongside fine-grained time-frequency localization. It integrates self-supervised feature pyramid matching, knowledge distillation, and multi-scale time-frequency anomaly scoring. Evaluated on mainstream benchmarks, our approach achieves an 8.2% absolute improvement in detection accuracy over supervised baselines. Moreover, it is the first to generate pixel-level, interpretable time-frequency anomaly heatmaps—enabling fully traceable, localized forensic analysis of synthetic speech artifacts.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Natural Language Processing: Speech
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The rise of AI-driven generative models has enabled the creation of highly realistic speech deepfakes - synthetic audio signals that can imitate target speakers' voices - raising critical security concerns. Existing methods for detecting speech deepfakes primarily rely on supervised learning, which suffers from two critical limitations: limited generalization to unseen synthesis techniques and a lack of explainability. In this paper, we address these issues by introducing a novel interpretable one-class detection framework, which reframes speech deepfake detection as an anomaly detection task. Our model is trained exclusively on real speech to characterize its distribution, enabling the classification of out-of-distribution samples as synthetically generated. Additionally, our framework produces interpretable anomaly maps during inference, highlighting anomalous regions across both time and frequency domains. This is done through a Student-Teacher Feature Pyramid Matching system, enhanced with Discrepancy Scaling to improve generalization capabilities across unseen data distributions. Extensive evaluations demonstrate the superior performance of our approach compared to the considered baselines, validating the effectiveness of framing speech deepfake detection as an anomaly detection problem.

Problem

• Detect and localize speech deepfakes via anomaly detection
• Overcome limitations of supervised learning in generalization
• Provide interpretable anomaly maps for time-frequency analysis

Innovation

• Interpretable one-class detection framework
• Student-Teacher Feature Pyramid Matching
• Discrepancy Scaling for generalization