AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

2026-06-01

AI Summary

Existing audio-visual speaker tracking methods suffer from limited performance in complex dynamic scenes, primarily due to their reliance on static co-occurrence assumptions and the absence of fine-grained, realistically challenging datasets. To address this gap, this work introduces AVTrack—the first human-centric Audio-Visual Instance Segmentation (AVIS) benchmark tailored for dynamic and complex environments, incorporating realistic challenges such as camera motion, occlusion, and speaker position changes. The dataset features multimodal fusion, instance-level annotations, and spatiotemporal alignment mechanisms. An accompanying evaluation protocol and baseline models reveal a significant performance drop compared to existing approaches, thereby validating the dataset’s difficulty and establishing a foundation for advancing high-order cross-modal spatiotemporal modeling research.

Abstract

Audio-visual speaker tracking aims to localize and track active speakers by leveraging auditory and visual cues, enabling fine-grained, human-centric scene understanding. This capability is essential for real-world applications such as intelligent video editing, surveillance, and human-computer interaction. However, existing datasets are largely limited to simple or homogeneous audio-visual scenes with coarse annotations. Such oversimplified settings bias evaluation toward static audio-visual co-occurrence, rather than rigorously assessing robust spatiotemporal modeling and cross-modal reasoning in complex, dynamic scenes. To address these limitations, we introduce AVTrack, a human-centric audio-visual instance segmentation (AVIS) dataset designed for dynamic real-world scenarios. AVTrack features diverse and challenging conditions, including camera motion, visual occlusions, and position changes. Evaluations of representative AVIS methods on AVTrack reveal substantial performance degradation, establishing AVTrack as a challenging benchmark for robust human-centric audio-visual scene understanding in complex environments. We further provide a simple yet effective baseline to facilitate future research. Project website: https://FudanCVL.github.io/AVTrack/

Problem

• audio-visual tracking
• complex scenes
• human-centric
• spatiotemporal modeling
• cross-modal reasoning

Innovation

• audio-visual tracking
• instance segmentation
• complex scenes
• cross-modal reasoning
• human-centric understanding