Can We Really Repurpose Multi-Speaker ASR Corpus for Speaker Diarization?

2025-07-12

AI Summary

This work investigates whether multi-speaker ASR corpora can effectively support speaker diarization. We find that the loosely defined speech segment boundaries in ASR data—often derived from automatic segmentation or transcription alignment—conflict with the strict boundary definitions required by diarization benchmarks, leading to inflated diarization error rates (DER), unreliable evaluation, and poor cross-dataset generalization. To address this, we propose a forced-alignment-based boundary normalization method that refines segment start/end points to conform to diarization conventions, coupled with lightweight post-processing to optimize neural diarization model training. Our approach significantly improves diarization performance: it reduces DER in both streaming and offline settings while simultaneously enhancing joint ASR accuracy. Crucially, this study is the first to systematically demonstrate that ASR segment boundary precision critically governs diarization generalization. The proposed framework establishes a reproducible technical pathway for cross-task data reuse in spoken language understanding.

Technology Category

• Natural Language Processing: Speech
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data integration and cleaning
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Neural speaker diarization is widely used for overlap-aware speaker diarization, but it requires large multi-speaker datasets for training. To meet this data requirement, large datasets are often constructed by combining multiple corpora, including those originally designed for multi-speaker automatic speech recognition (ASR). However, ASR datasets often feature loosely defined segment boundaries that do not align with the stricter conventions of diarization benchmarks. In this work, we show that such boundary looseness significantly impacts the diarization error rate, reducing evaluation reliability. We also reveal that models trained on data with varying boundary precision tend to learn dataset-specific looseness, leading to poor generalization across out-of-domain datasets. Training with standardized tight boundaries via forced alignment improves not only diarization performance, especially in streaming scenarios, but also ASR performance when combined with simple post-processing.

Problem

• Impact of loose segment boundaries on diarization error rate
• Poor generalization due to dataset-specific boundary looseness
• Improving diarization and ASR via standardized tight boundaries

Innovation

• Standardized tight boundaries via forced alignment
• Improves diarization in streaming scenarios
• Enhances ASR with simple post-processing