Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environments

2025-09-21

AI Summary

To address the lack of effective collaboration mechanisms for multi-agent audio-visual navigation (AVN) in dynamic 3D environments, this paper proposes MASTAVN—a novel multi-agent framework. MASTAVN introduces a cross-agent communication mechanism and a joint audio-visual fusion module, integrated within a scalable Transformer architecture and trained via multi-agent reinforcement learning. Evaluated on Replica and Matterport3D simulators, it achieves efficient spatiotemporal coordination during navigation. Compared to single-agent and non-collaborative baselines, MASTAVN significantly improves navigation success rates and reduces task completion time, demonstrating its efficacy in time-sensitive applications such as emergency response. Its core contributions are threefold: (1) the first dedicated multi-agent collaborative paradigm for AVN; (2) an end-to-end training strategy that jointly optimizes modality alignment and inter-agent communication; and (3) support for scalable, robust, decentralized navigation under partial observability and dynamic environmental constraints.

Technology Category

• Multiagent Systems: Multiagent Planning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Intelligent agents often require collaborative strategies to achieve complex tasks beyond individual capabilities in real-world scenarios. While existing audio-visual navigation (AVN) research mainly focuses on single-agent systems, their limitations emerge in dynamic 3D environments where rapid multi-agent coordination is critical, especially for time-sensitive applications like emergency response. This paper introduces MASTAVN (Multi-Agent Scalable Transformer Audio-Visual Navigation), a scalable framework enabling two agents to collaboratively localize and navigate toward an audio target in shared 3D environments. By integrating cross-agent communication protocols and joint audio-visual fusion mechanisms, MASTAVN enhances spatial reasoning and temporal synchronization. Through rigorous evaluation in photorealistic 3D simulators (Replica and Matterport3D), MASTAVN achieves significant reductions in task completion time and notable improvements in navigation success rates compared to single-agent and non-collaborative baselines. This highlights the essential role of spatiotemporal coordination in multi-agent systems. Our findings validate MASTAVN's effectiveness in time-sensitive emergency scenarios and establish a paradigm for advancing scalable multi-agent embodied intelligence in complex 3D environments.

Problem

• Enabling multi-agent collaboration for audio target navigation in 3D environments
• Overcoming single-agent limitations in dynamic audio-visual navigation tasks
• Enhancing spatiotemporal coordination for time-sensitive emergency response scenarios

Innovation

• Multi-agent collaboration for audio-visual navigation
• Cross-agent communication and joint fusion mechanisms
• Scalable transformer framework for 3D environments