Guided Speaker Embedding

2024-10-16

arXiv.org

AI Summary

To address the challenge of extracting target speaker embeddings from long, overlapping multi-speaker speech, this paper proposes an activity-guided end-to-end embedding extraction method. The core innovation lies in explicitly leveraging speaker activity (SA) as a guidance signal—introducing an activity-aware attention masking mechanism that performs conditional attention pooling directly on raw overlapped speech, thereby replacing the conventional paradigm reliant on pre-segmentation into clean single-speaker utterances. The method jointly models acoustic features and SA annotations using an ECAPA-TDNN backbone to learn robust, activity-conditioned speaker embeddings. Evaluated on speaker verification and speaker diarization tasks, the approach achieves significant performance gains under overlapping conditions, effectively overcoming the inherent limitations of traditional methods constrained to isolated single-speaker segments.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Active Learning
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

This paper proposes a guided speaker embedding extraction system, which extracts speaker embeddings of the target speaker using speech activities of target and interference speakers as clues. Several methods for long-form overlapped multi-speaker audio processing are typically two-staged: i) segment-level processing and ii) inter-segment speaker matching. Speaker embeddings are often used for the latter purpose. Typical speaker embedding extraction approaches only use single-speaker intervals to avoid corrupting the embeddings with speech from interference speakers. However, this often makes speaker embeddings impossible to extract because sufficiently long non-overlapping intervals are not always available. In this paper, we propose using speaker activities as clues to extract the embedding of the speaker-of-interest directly from overlapping speech. Specifically, we concatenate the activity of target and non-target speakers to acoustic features before being fed to the model. We also condition the attention weights used for pooling so that the attention weights of the intervals in which the target speaker is inactive are zero. The effectiveness of the proposed method is demonstrated in speaker verification and speaker diarization.

Problem

• Speaker Recognition
• Voice Separation
• Noisy Environment

Innovation

• Speaker Diarization
• Multi-Talker Environment
• Noise Robustness