BinauralVAE: Spatial Audio Reconstruction For World Models

2026-09-06

AI Summary

为解决视觉中心世界模型在特定环境下的局限，通过引入BinauralVAE技术，利用变分自编码器架构学习双耳信号的鲁棒潜在表示，增强空间音频重建能力。

Abstract

Embodied artificial intelligence has historically very much relied on visual perception, leading to a proliferation of multiple vision-centric world models. However, this reliance fails to capture spatial understanding in its entirety and can even present vulnerabilities in environments with visual occlusions, low-light conditions, or blackouts-scenarios, where acoustic information becomes a critical alternative for spatial awareness and navigation. Despite its potential, research into realistic spatial audio and particularly the development of audio-centric world models remains sparse. In this technical report, we introduce BinauralVAE: a flexible, open-source pipeline (https://github.com/Luizerko/BinauralVAE) that explores multiple models for spatialized audio reconstruction, progressing from fundamental baselines to advanced, mathematically grounded architectures. Our approach evaluates various Variational Autoencoder architectures -- including complex-valued variants -- to learn robust latent representations of binaural signals. Developed alongside AudioWorldSim, our methodology leverages realistic acoustic data captured as a simulated robot navigates an environment. This pipeline establishes a foundation for state representation in a future audio-based world model, designed to map the direct causal connection between navigational actions and their resulting acoustic consequences, and helping to enable sound as an essential complementary modality for spatial knowledge acquisition.

Problem

• spatial audio
• world models
• embodied artificial intelligence
• audio-centric

Innovation

• BinauralVAE
• Spatial Audio Reconstruction
• Variational Autoencoder
• Latent Representations
• Audio-centric World Models