Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Text

2026-02-12

AI Summary

This work addresses the limited robustness of end-to-end speech translation (E2E-ST) models against inflectional variations caused by non-native accents or dialects, a challenge exacerbated by the high cost of generating adversarial speech data. To overcome this, the authors propose the Cross-Modal Robustness Transfer (CMRT) framework, which— for the first time—enables adversarial robustness transfer from the text modality to the speech modality. Specifically, CMRT trains on inflectional adversarial examples constructed in the textual domain and effectively transfers the learned robustness to the speech translation model without requiring any adversarial speech data. Experimental results across four language pairs demonstrate that CMRT improves adversarial robustness by over 3 BLEU points on average, significantly outperforming existing approaches and establishing a new paradigm for efficiently building robust E2E-ST systems.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

End-to-End Speech Translation (E2E-ST) has seen significant advancements, yet current models are primarily benchmarked on curated,"clean"datasets. This overlooks critical real-world challenges, such as morphological robustness to inflectional variations common in non-native or dialectal speech. In this work, we adapt a text-based adversarial attack targeting inflectional morphology to the speech domain and demonstrate that state-of-the-art E2E-ST models are highly vulnerable it. While adversarial training effectively mitigates such risks in text-based tasks, generating high-quality adversarial speech data remains computationally expensive and technically challenging. To address this, we propose Cross-Modal Robustness Transfer (CMRT), a framework that transfers adversarial robustness from the text modality to the speech modality. Our method eliminates the requirement for adversarial speech data during training. Extensive experiments across four language pairs demonstrate that CMRT improves adversarial robustness by an average of more than 3 BLEU points, establishing a new baseline for robust E2E-ST without the overhead of generating adversarial speech.

Problem

• Speech Translation
• Adversarial Robustness
• Inflectional Morphology
• Cross-Modal Transfer
• End-to-End ST

Innovation

• Cross-Modal Robustness Transfer
• Adversarial Training
• Speech Translation
• Inflectional Morphology
• End-to-End ST