Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?

2026-06-24

AI Summary

This work addresses the structural misalignment between speech encoders and large language models caused by inconsistencies between language-specific and language-agnostic representations. To resolve this, the authors propose incorporating a translation objective into speech encoder pretraining, leveraging multilingual speech translation tasks to encourage the learning of language-independent speech representations. This study presents the first systematic demonstration of the critical role of translation-based pretraining in building effective speech-augmented large language models, establishing a new paradigm wherein translation-driven alignment enhances cross-lingual representation learning. Experimental results show that the proposed approach significantly improves performance across multiple downstream speech-language tasks, effectively strengthening both cross-modal fusion and linguistic generalization capabilities.

Abstract

Connecting a pre-trained speech encoder to a Large Language Model (LLM) is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM. Unlike encoders based on automatic speech recognition, which often produce representations in separate language-specific spaces, LLMs operate within a unified language-agnostic space. A mechanism is required to align the encoder's language-specific representations with the LLM's shared space. We argue that speech translation provides a principled way to achieve this. Unlike monolingual transcription, translation requires the model to bridge different languages and learn language-agnostic representations. We experimentally evaluate the impact of incorporating translation objectives into speech encoder pre-training. Our results demonstrate that translation-enhanced pre-training improves cross-modal integration and leads to superior performance across downstream Speech LLM tasks.

Problem

• speech encoder
• Large Language Model
• structural misalignment
• language-specific representations
• language-agnostic space

Innovation

• speech translation
• speech encoder pre-training
• language-agnostic representation
• Speech LLMs
• cross-modal alignment