Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation

2026-06-13

AI Summary

This study addresses the lack of effective evaluation and preservation mechanisms for cross-lingual lexical stress transfer in English-to-Chinese speech-to-speech translation (S2ST), particularly the absence of reliable automatic metrics for tonal languages. The authors construct the first English–Mandarin parallel speech dataset annotated with lexical stress, develop a Mandarin stress detector leveraging the XLS-R pretrained model, and integrate it with the English EmphAssess system to propose the first objective metric for stress evaluation in English-to-Chinese S2ST. Using this metric, they fine-tune CosyVoice3 to build a stress-aware S2ST system that significantly improves stress retention in translated speech while maintaining high translation quality. The proposed metric demonstrates strong correlation with human subjective judgments, confirming its validity and practical utility.

Abstract

Speech-to-speech translation (S2ST) systems have achieved impressive progress in semantic accuracy and speech naturalness. However, the cross-lingual transfer of lexical stress, a vital cue for emphasis and speaker intent, remains heavily underexplored, compounded by a lack of reliable automatic evaluation metrics for tonal languages like Chinese. We investigate English-to-Chinese S2ST stress transfer by constructing a stress-annotated Chinese dataset and an XLS-R-based Mandarin stress detector. Integrating this with the English EmphAssess system, we propose a novel objective metric for cross-lingual stress evaluation. Furthermore, we fine-tune CosyVoice3 to build a stress-aware S2ST system. Experiments demonstrate that our proposed S2ST architecture significantly outperforms existing systems in stress translation capability while maintaining competitive translation quality. Furthermore, our evaluation metric exhibits a strong correlation with human subjective judgments.

Problem

• lexical stress
• speech-to-speech translation
• cross-lingual transfer
• tonal languages
• automatic evaluation metrics

Innovation

• lexical stress
• speech-to-speech translation
• cross-lingual prosody
• objective evaluation metric
• stress-aware synthesis