BSM-iMagLS: ILD Informed Binaural Signal Matching for Reproduction with Head-Mounted Microphone Arrays

2025-01-30

AI Summary

Sparse and irregular microphone arrays in AR/VR headsets severely distort interaural level differences (ILDs), hindering high-fidelity binaural signal reconstruction. Method: This paper proposes an ILD-guided Binaural Signal Matching (BSM) framework. It innovatively embeds ILD constraints into magnitude least-squares (MagLS) optimization, yielding an ILD-aware MagLS (iMagLS) model. An end-to-end deep neural network solver is designed to jointly optimize magnitude spectra, ILDs, and magnitude derivatives. Robust HRTF modeling is further integrated to enhance spatial fidelity. Contribution/Results: Theoretical analysis, extensive simulations across multiple HRTFs and array configurations, and subjective listening tests demonstrate significant ILD error reduction, state-of-the-art magnitude reconstruction accuracy, and effective mitigation of spatial distortion in binaural rendering from sparse on-head arrays.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Headphone listening in applications such as augmented and virtual reality (AR and VR) relies on high-quality spatial audio to ensure immersion, making accurate binaural reproduction a critical component. As capture devices, wearable arrays with only a few microphones with irregular arrangement face challenges in achieving a reproduction quality comparable to that of arrays with a large number of microphones. Binaural signal matching (BSM) has recently been presented as a signal-independent approach for generating high-quality binaural signal using only a few microphones, which is further improved using magnitude-least squares (MagLS) optimization at high frequencies. This paper extends BSM with MagLS by introducing interaural level difference (ILD) into the MagLS, integrated into BSM (BSM-iMagLS). Using a deep neural network (DNN)-based solver, BSM-iMagLS achieves joint optimization of magnitude, ILD, and magnitude derivatives, improving spatial fidelity. Performance is validated through theoretical analysis, numerical simulations with diverse HRTFs and head-mounted array geometries, and listening experiments, demonstrating a substantial reduction in ILD errors while maintaining comparable magnitude accuracy to state-of-the-art solutions. The results highlight the potential of BSM-iMagLS to enhance binaural reproduction for wearable and portable devices.

Problem

• AR/VR
• 3D Audio
• Inter-Aural Level Difference (ILD)

Innovation

• BSM-iMagLS
• Deep Neural Network Optimization
• 3D Audio Rendering