Disentangling Speaker Traits for Deepfake Source Verification via Chebyshev Polynomial and Riemannian Metric Learning

2026-03-23

AI Summary

Existing deepfake audio provenance methods often assume that source embeddings are independent of speaker characteristics, yet this assumption remains unverified and speaker information may interfere with source attribution. This work presents the first systematic analysis of how speaker factors affect source verification and introduces a Speaker-Decoupled Metric Learning (SDML) framework. SDML explicitly disentangles speaker and generation-source features during optimization via a novel loss function, employs Chebyshev polynomials to mitigate gradient instability, and enhances source discriminability by incorporating Riemannian metrics in hyperbolic space. Evaluated on the MLAAD benchmark and four newly designed decoupling evaluation protocols, SDML significantly outperforms existing approaches, effectively eliminating speaker-induced interference and improving source verification accuracy.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Speech
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Speech deepfake source verification systems aims to determine whether two synthetic speech utterances originate from the same source generator, often assuming that the resulting source embeddings are independent of speaker traits. However, this assumption remains unverified. In this paper, we first investigate the impact of speaker factors on source verification. We propose a speaker-disentangled metric learning (SDML) framework incorporating two novel loss functions. The first leverages Chebyshev polynomial to mitigate gradient instability during disentanglement optimization. The second projects source and speaker embeddings into hyperbolic space, leveraging Riemannian metric distances to reduce speaker information and learn more discriminative source features. Experimental results on MLAAD benchmark, evaluated under four newly proposed protocols designed for source-speaker disentanglement scenarios, demonstrate the effectiveness of SDML framework. The code, evaluation protocols and demo website are available at https://github.com/xxuan-acoustics/RiemannSD-Net.

Problem

• deepfake source verification
• speaker disentanglement
• speech synthesis
• source attribution
• speaker traits

Innovation

• speaker disentanglement
• Chebyshev polynomial
• Riemannian metric learning
• deepfake source verification
• hyperbolic embedding