Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigation

2025-09-23

AI Summary

This work addresses the insufficient modeling of perceptual context in robot audio-visual collaborative navigation and sound source localization within dynamic environments. We propose an efficient multimodal navigation framework featuring a dynamic multi-object cross-modal fusion mechanism, integrated with a lightweight, context-aware enhanced Transformer architecture to enable selective alignment and deep semantic interaction between visual and auditory signals. Unlike existing approaches, our framework explicitly models spatiotemporal dependencies among heterogeneous sensory cues and supports real-time on-board sensor inference. Experiments on Replica and Matterport3D demonstrate state-of-the-art performance: +12.3% success rate, −18.7% path inefficiency, and superior cross-scene generalization—substantially improving robustness and adaptability in complex, dynamic settings.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Audiovisual embodied navigation enables robots to locate audio sources by dynamically integrating visual observations from onboard sensors with the auditory signals emitted by the target. The core challenge lies in effectively leveraging multimodal cues to guide navigation. While prior works have explored basic fusion of visual and audio data, they often overlook deeper perceptual context. To address this, we propose the Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigation (DMTF-AVN). Our approach uses a multi-target architecture coupled with a refined Transformer mechanism to filter and selectively fuse cross-modal information. Extensive experiments on the Replica and Matterport3D datasets demonstrate that DMTF-AVN achieves state-of-the-art performance, outperforming existing methods in success rate (SR), path efficiency (SPL), and scene adaptation (SNA). Furthermore, the model exhibits strong scalability and generalizability, paving the way for advanced multimodal fusion strategies in robotic navigation. The code and videos are available at https://github.com/zzzmmm-svg/DMTF.

Problem

• Dynamic fusion of visual and audio cues for navigation
• Filtering and selectively integrating cross-modal information
• Improving robot navigation success rate and path efficiency

Innovation

• Multi-target architecture for cross-modal fusion
• Refined Transformer mechanism for selective filtering
• Dynamic integration of audio-visual navigation cues