Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio

2025-05-16

AI Summary

Monaural multi-speaker automatic speech recognition (ASR) faces persistent challenges including data scarcity, ambiguous speaker attribution, and difficulty in overlapping speech recognition, compounded by the absence of a systematic survey on end-to-end (E2E) approaches. To address this gap, we propose the first unified taxonomy for monaural multi-speaker E2E-ASR, explicitly distinguishing and comparatively analyzing SIMO (single-input, multi-output) and SISO (single-input, single-output) architectural paradigms. We further introduce the novel “speaker-consistency hypothesis stitching” framework to enhance robustness in long-form speech modeling. Through comprehensive cross-benchmark evaluation on LibriCSS, AMI, and other datasets, we characterize performance boundaries and identify fundamental error sources across paradigms. Our analysis distills three critical open challenges—robustness, scalability, and real-time inference—providing both theoretical foundations and actionable technical pathways toward practical multi-speaker ASR systems.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Speech
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Monaural multi-speaker automatic speech recognition (ASR) remains challenging due to data scarcity and the intrinsic difficulty of recognizing and attributing words to individual speakers, particularly in overlapping speech. Recent advances have driven the shift from cascade systems to end-to-end (E2E) architectures, which reduce error propagation and better exploit the synergy between speech content and speaker identity. Despite rapid progress in E2E multi-speaker ASR, the field lacks a comprehensive review of recent developments. This survey provides a systematic taxonomy of E2E neural approaches for multi-speaker ASR, highlighting recent advances and comparative analysis. Specifically, we analyze: (1) architectural paradigms (SIMO vs.~SISO) for pre-segmented audio, analyzing their distinct characteristics and trade-offs; (2) recent architectural and algorithmic improvements based on these two paradigms; (3) extensions to long-form speech, including segmentation strategy and speaker-consistent hypothesis stitching. Further, we (4) evaluate and compare methods across standard benchmarks. We conclude with a discussion of open challenges and future research directions towards building robust and scalable multi-speaker ASR.

Problem

• Addressing data scarcity in monaural multi-speaker ASR
• Comparing E2E architectures for overlapping speech recognition
• Reviewing segmentation strategies for long-form speech ASR

Innovation

• End-to-end architectures reduce error propagation
• SIMO vs. SISO paradigms for pre-segmented audio
• Segmentation strategy for long-form speech