AuralNet: Hierarchical Attention-based 3D Binaural Localization of Overlapping Speakers

2025-06-03

AI Summary

This work addresses the problem of three-dimensional (3D) binaural sound source localization for overlapping speakers under noisy and reverberant conditions, without assuming prior knowledge of the number of sources. To this end, we propose a gated coarse-to-fine architecture. Methodologically, we introduce multi-head self-attention into 3D binaural signal modeling for the first time, integrate sector-based spatial partitioning with joint coarse classification and fine-grained regression, and design a masked multi-task loss function that enables adjustable spatial resolution and strong robustness. Compared to existing approaches, our method achieves significant improvements in both accuracy and stability for concurrent azimuth and elevation estimation. It attains state-of-the-art (SOTA) performance across diverse real-world noise–reverberation scenarios. The proposed framework establishes a new paradigm for calibration-free, low-latency 3D auditory perception.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: 3D Computer Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We propose AuralNet, a novel 3D multi-source binaural sound source localization approach that localizes overlapping sources in both azimuth and elevation without prior knowledge of the number of sources. AuralNet employs a gated coarse-tofine architecture, combining a coarse classification stage with a fine-grained regression stage, allowing for flexible spatial resolution through sector partitioning. The model incorporates a multi-head self-attention mechanism to capture spatial cues in binaural signals, enhancing robustness in noisy-reverberant environments. A masked multi-task loss function is designed to jointly optimize sound detection, azimuth, and elevation estimation. Extensive experiments in noisy-reverberant conditions demonstrate the superiority of AuralNet over recent methods

Problem

• Localizes overlapping sound sources in 3D space without prior source count
• Uses hierarchical attention to enhance robustness in noisy environments
• Jointly optimizes detection and spatial estimation via multi-task learning

Innovation

• Gated coarse-to-fine hierarchical architecture
• Multi-head self-attention for spatial cues
• Masked multi-task loss joint optimization