Target speaker anonymization in multi-speaker recordings

2025-10-10

AI Summary

This work addresses the underexplored problem of anonymizing only the target speaker—e.g., the customer in call-center dialogues—within multi-speaker conversational speech. We propose the first fine-grained target-speaker anonymization framework for multi-speaker audio. Our method integrates speaker separation, target-directed voiceprint feature perturbation, and context-aware speech reconstruction to precisely conceal the identity of the specified speaker. Innovatively, we design a joint privacy-utility evaluation metric to overcome the limitations of existing metrics in multi-speaker scenarios. Experiments demonstrate that our approach significantly enhances target-speaker identity protection—reducing privacy leakage by over 60%—while preserving speech intelligibility and conversational coherence. Degradations in speech quality (measured by PESQ) and ASR accuracy remain within acceptable bounds, confirming a favorable privacy–utility trade-off.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Speech
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Most of the existing speaker anonymization research has focused on single-speaker audio, leading to the development of techniques and evaluation metrics optimized for such condition. This study addresses the significant challenge of speaker anonymization within multi-speaker conversational audio, specifically when only a single target speaker needs to be anonymized. This scenario is highly relevant in contexts like call centers, where customer privacy necessitates anonymizing only the customer's voice in interactions with operators. Conventional anonymization methods are often not suitable for this task. Moreover, current evaluation methodology does not allow us to accurately assess privacy protection and utility in this complex multi-speaker scenario. This work aims to bridge these gaps by exploring effective strategies for targeted speaker anonymization in conversational audio, highlighting potential problems in their development and proposing corresponding improved evaluation methodologies.

Problem

• Anonymizing target speakers in multi-speaker conversational recordings
• Developing evaluation metrics for privacy and utility preservation
• Addressing limitations of conventional single-speaker anonymization methods

Innovation

• Anonymizes target speaker in multi-speaker recordings
• Develops strategies for conversational audio anonymization
• Proposes improved evaluation methodologies for privacy