CNN Models for Microphone Array Covariance Matrix Upsampling and Acoustic Imaging

2026-07-01

AI Summary

This work proposes a deep learning–based covariance matrix upsampling method to enhance the spatial resolution of acoustic imaging using a four-element tetrahedral microphone array under hardware-constrained conditions. The key innovation lies in the novel integration of frequency-dynamic convolution with a 2D convolutional neural network, which effectively captures the spatial–frequency coupling structure and frequency-dependent characteristics inherent in the covariance matrix. Experimental results demonstrate that the proposed approach reduces the root mean square error (RMSE) of sound source localization to 0.432, significantly outperforming a random baseline (RMSE = 0.548). Moreover, the reconstructed beamforming heatmaps closely approximate those produced by a ground-truth 32-channel spherical array, highlighting the method’s efficacy in high-fidelity acoustic imaging with minimal hardware resources.

Abstract

Acoustic imaging visualization is a core methodology in acoustics, enabling spatial analysis of sound sources and acoustic scenes. However, limited sensor availability in practical systems motivate approaches that enhance spatial resolution without increasing the hardware complexity. In this paper, we focus on upsampling virtually a tetrahedral 4-microphone array to a spherical 32-microphone array by estimating the covariance matrices of the channels employing deep learning techniques. Five neural network architectures are investigated for covariance upsampling for acoustic imaging using the real-world STARSS23 dataset. These models are developed to estimate a 32-microphone, time-frequency covariance matrix from a 4-microphone input covariance representation. The proposed architectures are based on 2D convolutional layers to capture the underlying spatial-spectral structure of covariance matrices, and are further enhanced with frequency dynamic convolution to model their frequency-dependent properties. The proposed architectures are evaluated in terms of root mean square error (RMSE) and using delay-and-sum beamforming acoustic imaging. Quantitative results show that all models outperform a random-guess baseline, which yields an RMSE of 0.548, with the best-performing architecture achieving an RMSE of 0.432. We analyze qualitatively the performance of the proposed models through beamforming heatmap visualizations derived from the 4-channel input covariance, the 32-channel ground truth, and the predicted 32-channel covariance matrices. These results demonstrate that covariance upsampling significantly enhances the effective performance of the 4-channel microphone array, producing sound maps that closely resemble those obtained with the 32-channel array.

Problem

• microphone array
• covariance matrix
• acoustic imaging
• spatial resolution
• upsampling

Innovation

• covariance matrix upsampling
• microphone array
• acoustic imaging
• frequency dynamic convolution
• deep learning