What Counts as an Error? Dual-Reference Benchmarking for Atypical ASR

2026-06-30

AI Summary

This work addresses the challenge of atypical speech recognition, where conventional evaluation practices relying on a single transcription standard conflate verbatim fidelity with semantic intent. To resolve this ambiguity, the study introduces the first dual-reference benchmark framework specifically designed for atypical speech such as stuttering, incorporating both verbatim transcripts (preserving repetitions and prolongations) and intent-based transcripts (normalized and redundancy-free). The authors systematically evaluate eleven state-of-the-art ASR models spanning encoder-decoder, CTC, and Transducer architectures under both reference types. Results reveal substantial discrepancies in model performance and ranking depending on the chosen transcription standard, demonstrating that the selection of evaluation criteria critically influences the real-world deployment of atypical ASR systems and providing empirical guidance for context-appropriate model selection.

Abstract

ASR systems have been often reported to underperform on atypical speech. An often conflated compounding factor is the existence of two valid transcription references: verbatim (actual produced speech, including repetitions/prolongations) and intended (the canonical form of the text with disfluencies removed) in atypical speech recognition depending on context and use-case. Most ASR evaluations conflate this duality into a single ground truth and reward systems that delete disfluencies, ignoring verbatim faithfulness. We benchmark 11 ASR models from encoder-decoder, CTC and transducer families using both verbatim and intended references on atypical stuttered speech as a case study. Our quantitative assessment underlines the disparity in model performance and rankings using the two transcript styles. Through this analysis, we highlight the importance of selecting a suitable transcription reference for valid model selection depending on the use-case, particularly for atypical ASR.

Problem

• atypical speech
• ASR evaluation
• verbatim transcription
• intended transcription
• disfluencies

Innovation

• dual-reference benchmarking
• atypical speech recognition
• verbatim transcription
• intended transcription
• disfluency handling