3D Audio-Visual Segmentation

2024-11-04

arXiv.org

AI Summary

This paper introduces 3D Audio-Visual Segmentation (3D AVS), a novel task that extends conventional 2D audio-visual segmentation by localizing and segmenting sound-emitting objects in 3D space conditioned on audio input. To support this task, we present 3DAVS-S34-O7—the first synthetic benchmark featuring spatial audio recordings and per-instance 3D semantic masks. We propose EchoSegNet, a model that synergistically integrates pretrained 2D audio-visual foundation models with explicit 3D scene representations. Its core innovations include a spatial-audio-guided mask alignment module and a refinement module that bridges 2D audio-visual cues with 3D geometry. Extensive experiments demonstrate significant improvements in 3D sound-source segmentation accuracy on 3DAVS-S34-O7. Our work establishes a foundational framework for 3D audio-aware perception, enabling applications in embodied AI, robotic manipulation, and immersive AR/VR systems requiring spatially grounded multimodal interaction.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Recognizing the sounding objects in scenes is a longstanding objective in embodied AI, with diverse applications in robotics and AR/VR/MR. To that end, Audio-Visual Segmentation (AVS), taking as condition an audio signal to identify the masks of the target sounding objects in an input image with synchronous camera and microphone sensors, has been recently advanced. However, this paradigm is still insufficient for real-world operation, as the mapping from 2D images to 3D scenes is missing. To address this fundamental limitation, we introduce a novel research problem, 3D Audio-Visual Segmentation, extending the existing AVS to the 3D output space. This problem poses more challenges due to variations in camera extrinsics, audio scattering, occlusions, and diverse acoustics across sounding object categories. To facilitate this research, we create the very first simulation based benchmark, 3DAVS-S34-O7, providing photorealistic 3D scene environments with grounded spatial audio under single-instance and multi-instance settings, across 34 scenes and 7 object categories. This is made possible by re-purposing the Habitat simulator to generate comprehensive annotations of sounding object locations and corresponding 3D masks. Subsequently, we propose a new approach, EchoSegnet, characterized by integrating the ready-to-use knowledge from pretrained 2D audio-visual foundation models synergistically with 3D visual scene representation through spatial audio-aware mask alignment and refinement. Extensive experiments demonstrate that EchoSegnet can effectively segment sounding objects in 3D space on our new benchmark, representing a significant advancement in the field of embodied AI. Project page: https://surrey-uplab.github.io/research/3d-audio-visual-segmentation/

Problem

• Extending audio-visual segmentation from 2D to 3D space
• Identifying sounding objects in 3D scenes using audio cues
• Addressing challenges in camera extrinsics and audio scattering

Innovation

• Extends AVS to 3D output space
• Uses Habitat simulator for spatial audio
• Integrates 2D foundation models with 3D representations