From General-Purpose Audio Tagging to Spatially Grounded Sound Event Localization and Detection

2026-06-26

AI Summary

This work addresses the challenge of effectively extending general-purpose audio tagging models to sound event localization and detection (SELD). The proposed AT2SELD framework integrates a pretrained audio semantic backbone with a lightweight first-order Ambisonics spatial processing module, optimizing the fusion of semantic and spatial information through multi-stage neural architecture search. Key innovations include a late-stage cross-coupling mechanism, residual spatial encoding, trajectory-level detection, Cartesian-coordinate-based direction-of-arrival (DOA) estimation, and permutation-aware supervision. Experiments demonstrate that the method achieves strong fixed-source localization performance and favorable transferability across multiple standard datasets. Further gains are attained by combining focal loss with activity-conditioned DOA supervision, while threshold calibration significantly enhances system reliability.

Abstract

This report investigates the extension of pretrained General-Purpose Audio Tagging (GP-AT) models toward spatially grounded Sound Event Localization and Detection (SELD). The proposed AT2SELD framework couples a pretrained AT backbone with compact First-Order Ambisonics (FOA) spatial processing, track-wise SED and Cartesian DOA estimation, permutation aware supervision, and calibration. It characterizes how semantic audio priors support localization-aware scene analysis under data, computation, and deployment constraints. The framework is developed through informed multi-stage Neural Architecture Search (NAS). Stage 1 shows that spectral FOA descriptors, based on magnitude, phase, and Intensity Vectors (IVs), provide the most reliable interface for semantic-to-spatial transfer. Stage 2 identifies early residual spatial encoding as the main capacity-sensitive component, while late track-wise abstraction and recurrent smoothing act mainly as refinement stages. Stage 3 shows that late cross-stitch coupling improves semantic-spatial interaction, whereas early fusion is costlier and less effective. Diagnostic evaluation analyzes the selected architecture under class balancing, focal loss, activity-conditioned DOA supervision, threshold calibration, and transfer across STARSS23, TAU2019, TAU-NIGENS2020, and TAU-NIGENS2021. Focal loss improves the activity point, active-only DOA supervision mitigates inactive target dominance, and validation-selected thresholds recover calibration without replacing spatial learning. Cross-dataset and oracle-activity analyses indicate strong fixed source localization on TAU2019, transferable representations from TAU NIGENS2021, and meaningful but uncertain behavior on STARSS23. Overall, GP-AT priors appear promising for SELD design when embedded in spatial-aware architectures and optimized through integrated calibration and deployment oriented strategies.

Problem

• Audio Tagging
• Sound Event Localization and Detection
• Spatial Audio
• Semantic-Spatial Transfer
• First-Order Ambisonics

Innovation

• Sound Event Localization and Detection
• General-Purpose Audio Tagging
• Neural Architecture Search
• First-Order Ambisonics
• Spatially Grounded Audio