Alethia: A Foundational Encoder for Voice Deepfakes

2026-04-30

AI Summary

Current deepfake audio detection methods rely on fine-tuning pretrained speech models and have approached performance saturation. This work proposes a novel self-supervised pretraining framework that introduces bottleneck masked embedding prediction and flow-matching-based spectrogram reconstruction to build the first general-purpose audio encoder explicitly designed for forgery-aware representation learning. By shifting from discrete reconstruction targets to continuous embedding prediction and generative modeling, the approach better captures subtle artifacts inherent in synthetic speech. Evaluated across five task categories and 56 benchmark datasets, the method significantly outperforms existing speech foundation models, demonstrating superior robustness and strong zero-shot generalization to unseen domains—such as singing voice forgery—without requiring task-specific adaptation.

Abstract

Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction. The outcome, Alethia, is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on $5$ different tasks with $56$ benchmark datasets, and note Alethia significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of continuous embedding prediction and generative pretraining for capturing deepfake artifacts.

Problem

• voice deepfakes
• detection
• localization
• speech foundation models
• pretraining

Innovation

• foundational audio encoder
• masked embedding prediction
• flow-matching
• voice deepfake detection
• generative pretraining