Attractor-Based Speech Separation of Multiple Utterances by Unknown Number of Speakers

2025-05-22

AI Summary

This paper addresses the challenges of unknown speaker count and multiple utterances per speaker in single-channel speech separation. We propose an attractor-driven end-to-end architecture that jointly models local and global temporal dependencies and introduces a learnable attractor module for dynamic source clustering. A lightweight, differentiable mechanism for estimating the number of active speakers is designed to simultaneously predict speaker count, perform separation, and detect speaker activity. To alleviate data scarcity, we construct a synthetic multi-speaker, multi-utterance dataset by augmenting LibriSpeech with WHAM! reverberation and noise. Experiments demonstrate significant improvements over baselines in speaker count estimation accuracy, separation quality (measured by SI-SNR improvement), and speaker activity detection (F1 score), particularly under reverberant and noisy conditions. The method exhibits strong generalization to both known and unknown numbers of speakers, confirming its robustness and practical applicability.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper addresses the problem of single-channel speech separation, where the number of speakers is unknown, and each speaker may speak multiple utterances. We propose a speech separation model that simultaneously performs separation, dynamically estimates the number of speakers, and detects individual speaker activities by integrating an attractor module. The proposed system outperforms existing methods by introducing an attractor-based architecture that effectively combines local and global temporal modeling for multi-utterance scenarios. To evaluate the method in reverberant and noisy conditions, a multi-speaker multi-utterance dataset was synthesized by combining Librispeech speech signals with WHAM! noise signals. The results demonstrate that the proposed system accurately estimates the number of sources. The system effectively detects source activities and separates the corresponding utterances into correct outputs in both known and unknown source count scenarios.

Problem

• Separate single-channel speech with unknown speaker count
• Dynamically estimate speakers and detect their activities
• Improve separation in reverberant and noisy conditions

Innovation

• Attractor module integrates separation and speaker counting
• Combines local and global temporal modeling effectively
• Accurate source count estimation in noisy conditions