Leveraging Sound Source Trajectories for Universal Sound Separation

2024-09-07

AI Summary

To address the performance limitations in mobile sound source separation caused by reliance on prior direction-of-arrival (DOA) information or inaccurate localization, this paper proposes a three-stage framework featuring joint optimization of separation and localization: (1) envelope-guided initial tracking, (2) iterative refinement via bidirectional mutual enhancement between separation and localization, and (3) neural beamforming-driven high-fidelity single-channel reconstruction. The core innovation lies in the first deep coupling mechanism between separation and localization—jointly modeling time-frequency masks and beamformer weights without requiring pre-specified DOAs—thereby significantly improving robustness in dynamic scenarios. Experiments under reverberant conditions demonstrate a 3.2 dB improvement in SI-SNR over baseline methods and a 41% reduction in DOA estimation error, with concurrent gains in both separation quality and trajectory accuracy.

Technology Category

• Search and Optimization: Local Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Existing methods utilizing spatial information for sound source separation require prior knowledge of the direction of arrival (DOA) of the source or utilize estimated but imprecise localization results, which impairs the separation performance, especially when the sound sources are moving. In fact, sound source localization and separation are interconnected problems, that is, sound source localization facilitates sound separation while sound separation contributes to refined source localization. This paper proposes a method utilizing the mutual facilitation mechanism between sound source localization and separation for moving sources. The proposed method comprises three stages. The first stage is initial tracking, which tracks each sound source from the audio mixture based on the source signal envelope estimation. These tracking results may lack sufficient accuracy. The second stage involves mutual facilitation: Sound separation is conducted using preliminary sound source tracking results. Subsequently, sound source tracking is performed on the separated signals, thereby refining the tracking precision. The refined trajectories further improve separation performance. This mutual facilitation process can be iterated multiple times. In the third stage, a neural beamformer estimates precise single-channel separation results based on the refined tracking trajectories and multi-channel separation outputs. Simulation experiments conducted under reverberant conditions and with moving sound sources demonstrate that the proposed method can achieve more accurate separation based on refined tracking results.

Problem

• Separating moving sound sources without prior DOA knowledge
• Improving sound localization and separation via mutual facilitation
• Enhancing separation accuracy with iterative refinement of trajectories

Innovation

• Utilizes mutual facilitation between localization and separation
• Iteratively refines tracking and separation performance
• Employs neural beamformer for precise single-channel results