Towards Localizing Conversation Partners using Head Motion

2026-04-26

AI Summary

This work addresses the challenge of accurately inferring a user’s auditory region of interest (ROI) in noisy environments, where existing spatial audio techniques often fail. The authors propose a non-invasive solution leveraging inertial measurement unit (IMU) data from smart glasses. They model head-motion cues to design HALo, a novel Auditory ROI Localization network, and enhance azimuth estimation accuracy by incorporating a prior on the number of concurrent speakers. Additionally, they introduce CoCo, the first IMU-only classifier for estimating the number of interlocutors. Experimental results demonstrate that HALo outperforms state-of-the-art methods by 21% in ROI localization accuracy, while CoCo achieves a classification accuracy of 0.74—representing a 35% improvement over the baseline—validating the system’s effectiveness in high-noise, multi-talker scenarios.

Technology Category

• Intelligent Robots: State Estimation
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Hardware-aware ML

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Many individuals struggle to understand conversation partners in noisy settings, particularly amid background speakers or due to hearing impairments. Emerging wearables like smartglasses offer a transformative opportunity to enhance speech from conversation partners. Crucial to this is identifying the direction in which the user wants to listen, which we refer to as the user's acoustic zones of interest. While current spatial audio-based methods can resolve the direction of vocal input, they are agnostic to listening preferences and have limited functionality in noisy settings with interfering speakers. To address this, behavioral cues are needed to actively infer a user's acoustic zones of interest. We explore the effectiveness of head-orienting behavior, captured by Inertial Measurement Units (IMUs) on smartglasses, as a modality for localizing these zones in seated conversations. We introduce HALo, a head-orientation-based acoustic zone localization network that leverages smartglasses' IMUs to non-invasively infer auditory zones of interest corresponding to conversation partner locations. By integrating an a priori estimate of the number of conversation partners, our approach yields a 21% performance improvement over existing methods. We complement this with CoCo, which classifies the number of conversation partners using only IMU data, achieving 0.74 accuracy and a 35% gain over rule-based and generic time-series baselines. We discuss practical considerations for feature extraction and inference and provide qualitative analyses over extended sessions. We also demonstrate a minimal end-to-end speech enhancement system, showing that head-orientation-based localization offers clear advantages in extremely noisy settings with multiple conversation partners.

Problem

• acoustic zones of interest
• conversation partner localization
• head motion
• noisy environments
• listening preference

Innovation

• head orientation
• acoustic zone of interest
• IMU-based localization
• speech enhancement
• smartglasses