Stereo Sound Event Localization and Detection with Onscreen/offscreen Classification

2025-07-16

AI Summary

This work addresses the Sound Event Localization and Detection (SELD) task in stereo audio, tackling its inherent azimuth ambiguity and challenges in distance estimation. It further introduces, for the first time, an “in-screen/out-of-screen” sound source classification subtask to accommodate field-of-view (FOV)-constrained audiovisual media scenarios. Methodologically, we propose a multimodal framework that jointly estimates direction-of-arrival (DOA) and distance from stereo signals while aligning with visual cues, thereby enabling sound event classification, 3D spatial localization (azimuth + distance), and FOV-aware discrimination. Compared to conventional omnidirectional SELD, our formulation extends both the task definition and evaluation metrics. A baseline system validated on stereo data demonstrates the feasibility and synergistic effectiveness of simultaneously optimizing classification, localization, and FOV classification—establishing a novel, lightweight paradigm for SELD tailored to real-world audiovisual content.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

This paper presents the objective, dataset, baseline, and metrics of Task 3 of the DCASE2025 Challenge on sound event localization and detection (SELD). In previous editions, the challenge used four-channel audio formats of first-order Ambisonics (FOA) and microphone array. In contrast, this year's challenge investigates SELD with stereo audio data (termed stereo SELD). This change shifts the focus from more specialized 360° audio and audiovisual scene analysis to more commonplace audio and media scenarios with limited field-of-view (FOV). Due to inherent angular ambiguities in stereo audio data, the task focuses on direction-of-arrival (DOA) estimation in the azimuth plane (left-right axis) along with distance estimation. The challenge remains divided into two tracks: audio-only and audiovisual, with the audiovisual track introducing a new sub-task of onscreen/offscreen event classification necessitated by the limited FOV. This challenge introduces the DCASE2025 Task3 Stereo SELD Dataset, whose stereo audio and perspective video clips are sampled and converted from the STARSS23 recordings. The baseline system is designed to process stereo audio and corresponding video frames as inputs. In addition to the typical SELD event classification and localization, it integrates onscreen/offscreen classification for the audiovisual track. The evaluation metrics have been modified to introduce an onscreen/offscreen accuracy metric, which assesses the models' ability to identify which sound sources are onscreen. In the experimental evaluation, the baseline system performs reasonably well with the stereo audio data.

Problem

• Stereo audio-based sound event localization and detection (SELD).
• Direction-of-arrival and distance estimation with stereo data.
• Onscreen/offscreen classification for limited field-of-view scenarios.

Innovation

• Stereo audio data for SELD
• Onscreen/offscreen event classification
• Modified evaluation metrics