Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios

2025-05-30

AI Summary

Low-resource and zero-resource speech-to-text translation (S2TT) suffers from a severe scarcity of labeled speech data. Method: We propose a phoneme-augmented chain-of-thought (CoT) reasoning framework that leverages phoneme recognition as an interpretable intermediate representation—marking the first integration of phonemic representations into the CoT paradigm for unsupervised cross-lingual transfer. Our approach jointly models speech, phonemes, and text in three stages, synergistically combining multilingual large language models (MLLMs) with a progressive curriculum learning strategy. Contribution/Results: The method achieves substantial improvements in translation quality for low-resource languages across multilingual S2TT benchmarks. Notably, it enables the first end-to-end S2TT for zero-resource languages without any target-language speech or text supervision. It demonstrates strong generalization capability and practical deployability, bridging a critical gap between interpretability, unsupervised adaptation, and real-world applicability in speech translation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We propose a Speech-to-Text Translation (S2TT) approach that integrates phoneme representations into a Chain-of-Thought (CoT) framework to improve translation in low-resource and zero-resource settings. By introducing phoneme recognition as an intermediate step, we enhance cross-lingual transfer, enabling translation even for languages with no labeled speech data. Our system builds on a multilingual LLM, which we extend to process speech and phonemes. Training follows a curriculum learning strategy that progressively introduces more complex tasks. Experiments on multilingual S2TT benchmarks show that phoneme-augmented CoT improves translation quality in low-resource conditions and enables zero-resource translation, while slightly impacting high-resource performance. Despite this trade-off, our findings demonstrate that phoneme-based CoT is a promising step toward making S2TT more accessible across diverse languages.

Problem

• Improving speech-to-text translation in low-resource settings
• Enabling zero-resource translation using phoneme-augmented CoT
• Enhancing cross-lingual transfer with phoneme recognition

Innovation

• Integrates phoneme representations into CoT
• Uses phoneme recognition for cross-lingual transfer
• Employs curriculum learning for progressive training