When Multiple Scripts Matter: Evaluating ASR in Clinical Settings

2026-06-16

AI Summary

This study addresses the underestimation of automatic speech recognition (ASR) performance in non-English clinical settings, where valid orthographic variants of the same term—written in multiple scripts—are erroneously penalized by conventional string-matching evaluation metrics. To remedy this, the authors propose MultiClin, the first multi-script evaluation framework tailored for clinical ASR, which employs multi-reference assessment to more fairly measure model robustness to orthographic variation. The work systematically investigates the impact of script consistency in training data, revealing that script unification substantially improves recognition accuracy, while a 50% script-mixing ratio induces maximal model uncertainty. Experiments demonstrate that the proposed multi-script-aware evaluation significantly enhances the fidelity of performance measurement, and that script normalization consistently yields optimal results across diverse ASR architectures. The dataset and code are publicly released.

Abstract

Automatic speech recognition (ASR) in non-English clinical settings is challenged by multiscript variability, where the same term may appear in multiple valid orthographic forms. Conventional string-matching evaluation metrics often underestimate ASR performance by treating orthographic variants as errors. To address this issue, we introduce MultiClin, a clinical ASR benchmark designed to evaluate robustness to multiscript variability. Experiments across diverse ASR models show that multiscript-aware evaluation provides a fairer assessment of recognition quality than conventional single-reference evaluation. We further investigate the impact of script consistency during training and find that inconsistent script mappings increase orthographic uncertainty and hinder model convergence, with a balanced 50% mapping ratio producing the highest entropy. In contrast, script unification consistently yields the best ASR performance. Our dataset and code are publicly available at: https://github.com/aitrics-ronaldo/Interspeech_MultiClin.

Problem

• multiscript variability
• automatic speech recognition
• clinical ASR
• orthographic variants
• evaluation metrics

Innovation

• multiscript variability
• clinical ASR
• script unification
• orthographic uncertainty
• MultiClin benchmark