SpeechCompass: Enhancing Mobile Captioning with Diarization and Directional Guidance via Multi-Microphone Localization

2025-02-12

AI Summary

To address the inability of mobile speech-to-text systems to identify speaker direction in group conversations—resulting in poor intelligibility and low interaction efficiency—this paper proposes the first lightweight, on-device real-time solution integrating sound source localization (SSL) and speaker diarization. Methodologically, it fuses multi-microphone array signal processing, time-difference-of-arrival (TDOA)-based localization, and embedded microcontroller deployment, while introducing a low-power directional visualization interface (e.g., dynamic azimuth arrows) tailored for hearing-impaired users. The key contribution is the first tight integration of real-time SSL and diarization within a mobile speech transcription pipeline. Technical evaluation demonstrates millisecond-level localization latency and accuracy; eight frequent users unanimously affirmed the utility of directional guidance; and 97% of participants reported a significant improvement in group conversation transcription intelligibility and interaction efficiency.

Technology Category

• Natural Language Processing: Speech
• Intelligent Robots: Localization, Mapping, and Navigation
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Speech-to-text capabilities on mobile devices have proven helpful for hearing and speech accessibility, language translation, note-taking, and meeting transcripts. However, our foundational large-scale survey (n=263) shows that the inability to distinguish and indicate speaker direction makes them challenging in group conversations. SpeechCompass addresses this limitation through real-time, multi-microphone speech localization, where the direction of speech allows visual separation and guidance (e.g., arrows) in the user interface. We introduce efficient real-time audio localization algorithms and custom sound perception hardware running on a low-power microcontroller and four integrated microphones, which we characterize in technical evaluations. Informed by a large-scale survey (n=494), we conducted an in-person study of group conversations with eight frequent users of mobile speech-to-text, who provided feedback on five visualization styles. The value of diarization and visualizing localization was consistent across participants, with everyone agreeing on the value and potential of directional guidance for group conversations.

Problem

• Enhances mobile captioning with speaker direction
• Improves group conversation accessibility
• Integrates real-time multi-microphone localization

Innovation

• Multi-microphone speech localization
• Real-time audio algorithms
• Directional guidance visualization