Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

2026-08-17

AI Summary

This study addresses the limited robustness of cross-modal alignment in existing audio-visual segmentation methods caused by insufficient geometric cues. To overcome this, we propose DGCM-AVS, a tri-modal framework that pioneeringly incorporates depth as an independent modality. Specifically, we design a depth-aware dynamic modulator and a progressive fusion mechanism to jointly model audio, visual, and depth information for precise sounding object localization. Experiments on the AVSS dataset demonstrate that our method achieves relative improvements of 10.2% in m_J and 8.7% in m_F, significantly outperforming state-of-the-art approaches. These results effectively validate the critical role of spatial structural cues in enhancing cross-modal alignment accuracy for audio-visual segmentation tasks.

Abstract

Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human-computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction.

Problem

• Audio-Visual Segmentation
• Geometric Cues
• Cross-modal Alignment
• Depth Information

Innovation

• Depth-Guided Collaborative Modeling
• Audio-Visual Segmentation
• Depth-Aware Dynamic Modulator
• Progressive Fusion
• Tri-modal Framework