Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments

2025-02-23

AI Summary

This paper addresses the challenging problem of target speaker extraction from multi-speaker overlapping audio without access to clean reference speech. To this end, we propose a novel paradigm based on contrastive learning between noisy but semantically distinct positive (“target speaking”) and negative (“target not speaking”) segments under noisy conditions. Our method employs an end-to-end differentiable source separation network and introduces a dedicated pretraining strategy to enhance robustness. Crucially, it eliminates the reliance on clean enrollment utterances—marking the first such approach—thereby significantly improving practical applicability in real-world scenarios. Evaluated on diverse highly overlapping settings—including the cocktail party problem—our method achieves state-of-the-art performance and demonstrates strong generalization across unseen noise types and speaker identities. This work provides a principled, reference-free framework for speaker extraction in realistic, acoustically degraded environments.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Target speaker extraction focuses on isolating a specific speaker's voice from an audio mixture containing multiple speakers. To provide information about the target speaker's identity, prior works have utilized clean audio examples as conditioning inputs. However, such clean audio examples are not always readily available (e.g. It is impractical to obtain a clean audio example of a stranger's voice at a cocktail party without stepping away from the noisy environment). Limited prior research has explored extracting the target speaker's characteristics from noisy audio examples, which may include overlapping speech from disturbing speakers. In this work, we focus on target speaker extraction when multiple speakers are present during the enrollment stage, through leveraging differences between audio segments where the target speakers are speaking (Positive Enrollments) and segments where they are not (Negative Enrollments). Experiments show the effectiveness of our model architecture and the dedicated pretraining method for the proposed task. Our method achieves state-of-the-art performance in the proposed application settings and demonstrates strong generalizability across challenging and realistic scenarios.

Problem

• Extracting target speaker from noisy audio mixtures.
• Utilizing noisy positive and negative audio enrollments.
• Improving speaker extraction in multi-speaker environments.

Innovation

• Noisy audio enrollment comparison
• Positive and negative segment analysis
• State-of-the-art extraction performance