Speaker head orientation estimation with a single microphone array using phase spectrogram features

2026-07-02

AI Summary

This work addresses the problem of speaker head orientation estimation using a single microphone array by proposing a deep learning approach based on short-time Fourier transform phase spectra. The method uniquely integrates phase spectral features with a hybrid neural architecture that combines convolutional, recurrent, and self-attention mechanisms, eliminating the need for handcrafted physical features or raw waveforms. Trained on large-scale simulated data and fine-tuned on real-world recordings, the model achieves state-of-the-art performance in both clean and noisy conditions. After personalized adaptation, it attains an average angular error as low as 11.3 degrees, significantly enhancing cross-domain robustness and estimation accuracy.

Abstract

Estimating a speaker's head orientation from audio can provide valuable information in smart environments, meetings, and driver monitoring. We propose a novel approach that leverages the phase component of the short-time Fourier transform from a single microphone array as input to a deep neural network combining convolutional, recurrent, and self-attention layers. Unlike prior methods that use physics-informed handcrafted features or raw waveform inputs, our approach enables robust learning from simulated and real data. Trained on a large-scale dataset generated with voice directivity patterns and fine-tuned on real recordings, our model achieves state-of-the-art accuracy, outperforming baselines under both clean and noisy conditions. Personalization experiments further demonstrate significant gains, reaching a mean angular error of 11.3 degrees when adapting to individual users and environments.

Problem

• head orientation estimation
• microphone array
• phase spectrogram
• speaker localization
• audio-based sensing

Innovation

• phase spectrogram
• head orientation estimation
• single microphone array
• deep neural network
• self-attention