A Hybrid Mamba for Audio-Visual Navigation

2026-07-14

AI Summary

This work addresses the limitations of conventional CNN-RNN architectures in effectively modeling dynamic multimodal sequences for audio-visual navigation, which hinder the capture of global time-frequency dependencies and long-range temporal aggregation. To overcome this, the study introduces Samba, a novel hybrid architecture that incorporates Mamba for the first time in this domain. Specifically, a Mamba State Encoder (M-SE) replaces the GRU to enable adaptive temporal modeling, while an Audio Mamba Encoder (AME) is designed to capture global time-frequency dependencies in spectrograms. The proposed approach breaks through the bottlenecks of traditional paradigms, achieving an 11.3% improvement in navigation success rate on Matterport3D and even more pronounced gains on the finer-grained Replica dataset, along with enhanced generalization capability and reduced computational overhead.

Abstract

Since the paradigm centered on convolutional neural networks and recurrent architectures was established in 2020, the fundamental backbone networks for audio-visual navigation have undergone no essential changes for more than five years, making them inadequate to support efficient representation of dynamic multimodal sequences. This paper proposes Samba(A Hybrid Mamba for Audio-Visual Navigation). It uses the adaptive selection-enabled Mamba State Encoder (M-SE) to replace conventional GRUs for temporal aggregation, and constructs an Audio Mamba Encoder (AME) to remedy the limitations of convolutional operators in capturing global time-frequency dependencies in spectrograms. Experiments demonstrate that Samba exhibits exceptional generalization performance when facing unheard sound sources and unseen scenes. On the Matterport3D dataset, it improves the navigation success rate (SR) by 11.3\% compared with existing state-of-the-art models, and the performance gain is even more pronounced on the Replica dataset, which features finer scene structures. Such modernized architectural reconstruction unlocks stronger embodied representation capabilities at a lower computational cost, thereby providing a highly robust technical pathway for paradigm evolution in the field of audio-visual navigation.

Problem

• audio-visual navigation
• multimodal sequences
• temporal aggregation
• global time-frequency dependencies
• embodied representation

Innovation

• Mamba
• Audio-Visual Navigation
• Temporal Aggregation
• Global Time-Frequency Dependencies
• Embodied Representation