Problem
Research questions and friction points this paper is trying to address.
Audio-Visual Navigation
binaural audio reliability
unheard sound generalization
cross-modal fusion
acoustic environment
Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation
🤖 AI Summary
This work addresses the challenge that unseen sound source categories in complex acoustic environments render binaural audio cues unreliable, significantly degrading audio-visual navigation performance. To mitigate this issue, the authors propose a Reliability-Aware Audio-Visual Navigation (RAVN) framework, which employs an unsupervised Acoustic Geometry Reasoner (AGR) to estimate the reliability of audio cues. A Reliability-Aware Geometric Modulation (RAGM) mechanism is further introduced to dynamically gate visual features in a soft manner, thereby alleviating cross-modal conflicts. The method is trained using heteroscedastic Gaussian negative log-likelihood, enabling robust generalization to unheard sounds. Experiments on SoundSpaces with Replica and Matterport3D scenes demonstrate substantial improvements in both generalization capability and navigation robustness under previously unencountered acoustic conditions.
📝 Abstract
Audio-Visual Navigation (AVN) requires an embodied agent to navigate toward a sound source by utilizing both vision and binaural audio. A core challenge arises in complex acoustic environments, where binaural cues become intermittently unreliable, particularly when generalizing to previously unheard sound categories. To address this, we propose RAVN (Reliability-Aware Audio-Visual Navigation), a framework that conditions cross-modal fusion on audio-derived reliability cues, dynamically calibrating the integration of audio and visual inputs. RAVN introduces an Acoustic Geometry Reasoner (AGR) that is trained with geometric proxy supervision. Using a heteroscedastic Gaussian NLL objective, AGR learns observation-dependent dispersion as a practical reliability cue, eliminating the need for geometric labels during inference. Additionally, we introduce Reliability-Aware Geometric Modulation (RAGM), which converts the learned cue into a soft gate to modulate visual features, thereby mitigating cross-modal conflicts. We evaluate RAVN on SoundSpaces using both Replica and Matterport3D environments, and the results show consistent improvements in navigation performance, with notable robustness in the challenging unheard sound setting.
Innovation
Methods, ideas, or system contributions that make the work stand out.
reliability-aware fusion
audio-visual navigation
acoustic geometry reasoner
heteroscedastic uncertainty
cross-modal modulation
🔎 Similar Papers
No similar papers found.
