Efficient Training for Cross-lingual Speech Language Models

2026-04-13

AI Summary

This work addresses the limitations of current speech large language models in cross-lingual generalization, which stem from scarce speech data and challenges in multimodal alignment. The authors propose CSLM, a framework built upon discrete speech tokens that leverages continual pretraining to achieve efficient cross-lingual and cross-modal alignment. To enhance generation quality and response latency, they introduce a speech-text interleaved instruction tuning strategy. Notably, CSLM does not rely on massive amounts of speech data, substantially improving language scalability and fine-grained alignment capabilities. The model demonstrates superior generalization and alignment performance across multimodal, monolingual, and cross-lingual dialogue tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Currently, large language models (LLMs) predominantly focus on the text modality. To enable more natural human-AI interaction, speech LLMs are emerging, but building effective end-to-end speech LLMs remains challenging due to limited data and the difficulty in expanding to more languages. In this paper, we introduce Cross-lingual Speech Language Model (CSLM), an efficient training method for cross-lingual speech LLMs based on discrete speech tokens. We propose a novel alignment strategy that achieves cross-modal and cross-lingual alignment through continual pre-training. By conducting instruction fine-tuning following a speech-text interleaved chain-of-modality generation process, we enhance modal alignment at a finer granularity, thereby improving generation quality and reducing latency. CSLM aligns different modalities and languages simultaneously without the need for massive speech data, thus exhibiting good language scalability. Evaluations on cross-modal tasks, mono-lingual conversational tasks, and cross-lingual conversational tasks demonstrate CSLM's strong cross-modal alignment capabilities and general task abilities. (Code is available at: https://github.com/ictnlp/CSLM)

Problem

• speech language models
• cross-lingual
• efficient training
• limited data
• language scalability

Innovation

• cross-lingual
• speech language model
• discrete speech tokens
• modality alignment
• continual pre-training