Scores Know Bobs Voice: Speaker Impersonation Attack

2026-03-03

AI Summary

This work addresses the inefficiency of existing score-based voice imitation attacks, which suffer from suboptimal optimization in the raw waveform space and geometric misalignment between the latent space of generative models and the discriminative feature space of speaker recognition systems, leading to biased attack directions. To overcome this, the authors propose an inversion-based generative attack framework that, for the first time, achieves geometric alignment between the generative model’s latent space and the speaker embedding space through a feature-aligned inversion strategy. This alignment enables efficient attack paradigms such as subspace projection. Experimental results demonstrate that the proposed method achieves comparable attack success rates with only one-tenth the query budget of prior approaches; notably, the subspace projection variant attains a 91.65% success rate within just 50 queries.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Security and Privacy: Large-scale security measurements

Abstract

Advances in deep learning have enabled the widespread deployment of speaker recognition systems (SRSs), yet they remain vulnerable to score-based impersonation attacks. Existing attacks that operate directly on raw waveforms require a large number of queries due to the difficulty of optimizing in high-dimensional audio spaces. Latent-space optimization within generative models offers improved efficiency, but these latent spaces are shaped by data distribution matching and do not inherently capture speaker-discriminative geometry. As a result, optimization trajectories often fail to align with the adversarial direction needed to maximize victim scores. To address this limitation, we propose an inversion-based generative attack framework that explicitly aligns the latent space of the synthesis model with the discriminative feature space of SRSs. We first analyze the requirements of an inverse model for score-based attacks and introduce a feature-aligned inversion strategy that geometrically synchronizes latent representations with speaker embeddings. This alignment ensures that latent updates directly translate into score improvements. Moreover, it enables new attack paradigms, including subspace-projection-based attacks, which were previously infeasible due to the absence of a faithful feature-to-audio mapping. Experiments show that our method significantly improves query efficiency, achieving competitive attack success rates with on average 10x fewer queries than prior approaches. In particular, the enabled subspace-projection-based attack attains up to 91.65% success using only 50 queries. These findings establish feature-aligned inversion as a key tool for evaluating the robustness of modern SRSs against score-based impersonation threats.

Problem

• speaker impersonation
• score-based attack
• latent space
• speaker recognition
• adversarial attack

Innovation

• feature-aligned inversion
• speaker impersonation attack
• latent space optimization
• score-based attack
• subspace-projection attack