Multi-Channel Differential ASR for Robust Wearer Speech Recognition on Smart Glasses

2025-09-17

AI Summary

Smart glasses suffer from severe degradation in wearer speech recognition (WSR) performance and error propagation in downstream NLP tasks due to interference from bystander speech (side-talk) in real-world environments. To address this, we propose a multi-channel differential automatic speech recognition (ASR) framework. Our method innovatively integrates beamforming, dynamic microphone selection, and a lightweight lateral speech detection model to generate robust differential signals at the front end. Moreover, we are the first to embed the differential mechanism into the end-to-end ASR joint optimization pipeline, enabling co-optimized interference suppression and speech recognition. Evaluated on both simulated and real-world datasets, our framework achieves up to a 18.0% relative reduction in word error rate (WER), significantly enhancing the stability and practicality of WSR under challenging acoustic conditions.

Technology Category

• Natural Language Processing: Speech
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Hardware-aware ML

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

With the growing adoption of wearable devices such as smart glasses for AI assistants, wearer speech recognition (WSR) is becoming increasingly critical to next-generation human-computer interfaces. However, in real environments, interference from side-talk speech remains a significant challenge to WSR and may cause accumulated errors for downstream tasks such as natural language processing. In this work, we introduce a novel multi-channel differential automatic speech recognition (ASR) method for robust WSR on smart glasses. The proposed system takes differential inputs from different frontends that complement each other to improve the robustness of WSR, including a beamformer, microphone selection, and a lightweight side-talk detection model. Evaluations on both simulated and real datasets demonstrate that the proposed system outperforms the traditional approach, achieving up to an 18.0% relative reduction in word error rate.

Problem

• Robust wearer speech recognition on smart glasses
• Addressing side-talk interference in speech recognition
• Improving accuracy through multi-channel differential ASR

Innovation

• Multi-channel differential ASR system
• Complementary frontends with beamformer and selection
• Lightweight side-talk detection model