Universal Semantic Disentangled Privacy-preserving Speech Representation Learning

2025-05-19

AI Summary

Speech LLM training poses significant risks of speaker identity leakage. To address this, we propose the Universal Speech Codec (USC), the first framework to explicitly decouple semantic content from privacy-sensitive acoustic features via a dual-stream semantic-acoustic architecture. USC jointly optimizes three objectives: semantic fidelity, preservation of paralinguistic attributes (e.g., prosody and emotion), and speaker anonymization. We introduce a privacy-aware reconstruction loss and adversarial de-identification training, and establish a novel perceptual-test-based privacy evaluation paradigm. Experiments demonstrate that USC achieves state-of-the-art performance on downstream tasks—including automatic speech recognition and emotion recognition—while outperforming mainstream speech codecs in reconstruction quality. Crucially, USC reduces success rates of speaker identity attacks by a substantial margin, thereby reconciling high-fidelity representation learning with rigorous privacy protection.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

The use of audio recordings of human speech to train LLMs poses privacy concerns due to these models' potential to generate outputs that closely resemble artifacts in the training data. In this study, we propose a speaker privacy-preserving representation learning method through the Universal Speech Codec (USC), a computationally efficient encoder-decoder model that disentangles speech into: (i) privacy-preserving semantically rich representations, capturing content and speech paralinguistics, and (ii) residual acoustic and speaker representations that enables high-fidelity reconstruction. Extensive evaluations presented show that USC's semantic representation preserves content, prosody, and sentiment, while removing potentially identifiable speaker attributes. Combining both representations, USC achieves state-of-the-art speech reconstruction. Additionally, we introduce an evaluation methodology for measuring privacy-preserving properties, aligning with perceptual tests. We compare USC against other codecs in the literature and demonstrate its effectiveness on privacy-preserving representation learning, illustrating the trade-offs of speaker anonymization, paralinguistics retention and content preservation in the learned semantic representations. Audio samples are shared in https://www.amazon.science/usc-samples.

Problem

• Privacy risks in speech data for LLM training
• Disentangling speech into secure and identifiable components
• Balancing anonymization with content and paralinguistic retention

Innovation

• Universal Speech Codec disentangles speech representations
• Semantic representation removes identifiable speaker attributes
• State-of-the-art speech reconstruction with privacy preservation