🤖 AI Summary
This study addresses a critical limitation of ASR-roundtrip evaluation in Chinese news text-to-speech (TTS) systems, where context- or domain-convention-dependent reading errors are often masked, leading to perceptible errors being misjudged as correct. The work presents the first systematic analysis of this blind spot and introduces a segment-isolation diagnostic technique that combines human annotation with multi-system comparisons—employing MiMo and CosyVoice TTS systems alongside Qwen3-ASR and Paraformer ASR—to re-expose previously concealed errors. Among 110 high-risk cases, 97 were confirmed as actual errors; Qwen3-ASR superficially recovered 40 of them, whereas Paraformer recovered only 2. These findings demonstrate that ASR-roundtrip evaluation alone is insufficient for assessing Chinese news TTS quality and underscore the necessity of context-sensitive error detection.
📝 Abstract
ASR-roundtrip evaluation is widely used as a scalable proxy for text-to-speech (TTS) intelligibility, but it can produce false negatives for reading errors perceived by listeners. We study Chinese news TTS spans whose correct reading depends on context or domain conventions, such as sports scores, aircraft models, technical units, and membership names. In these cases, Raw TTS can choose a plausible but wrong reading while ASR transcribes the audio as the intended or surface-correct text. A targeted audit over 110 high-risk MiMo TTS cases, reported with a complete denominator, confirms 46 masked false negatives, 9 exposed TTS errors, and 55 cases with no Raw TTS error. A span-isolation diagnostic re-exposes 18/46 previously masked errors. A Raw-only CosyVoice audit on the same targeted pool confirms 51 masked cases. Across the 97 TTS-specific audio files labeled confirmed masked across the two audits, Qwen3-ASR surface-recovers 40 cases, whereas Paraformer does so in only 2. The results suggest that ASR-roundtrip is useful for screening but insufficient as standalone ground truth for Chinese news reading-risk evaluation.