🤖 AI Summary
This work addresses key challenges in speech recognition evaluation under scenarios involving multiple reference transcriptions, non-Latin scripts, long-form audio, and streaming conditions. The authors propose an enhanced string alignment algorithm that accommodates multiple references and arbitrary-length insertions, enabling more accurate assessment of automatic speech recognition (ASR) systems. They construct DiverseSpeech-Ru, a high-quality Russian long-form multi-reference test set, and develop a unified evaluation and visualization toolkit supporting both streaming and offline ASR paradigms. Their approach not only uncovers model overfitting to specific transcription styles but also substantially improves the accuracy and interpretability of evaluations for multi-reference and streaming ASR systems.
📝 Abstract
We propose several improvements to the speech recognition evaluation. First, we propose a string alignment algorithm that supports both multi-reference labeling, arbitrary-length insertions and better word alignment. This is especially useful for non-Latin languages, those with rich word formation, to label cluttered or longform speech. Secondly, we collect a novel test set DiverseSpeech-Ru of longform in-the-wild Russian speech with careful multi-reference labeling. We also perform multi-reference relabeling of popular Russian tests set and study fine-tuning dynamics on its corresponding train set. We demonstrate that the model often adopts to dataset-specific labeling, causing an illusion of metric improvement. Based on the improved word alignment, we develop tools to evaluate streaming speech recognition and to align multiple transcriptions to compare them visually. Additionally, we provide uniform wrappers for many offline and streaming speech recognition models. Our code will be made publicly available.