Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

2026-07-02

AI Summary

This work addresses the challenges of data scarcity and sequence-length bottlenecks in instruction tuning for spoken language models, which typically rely on large-scale speech instruction datasets. The authors propose SpeechCombine, a method that transfers the instruction-following capabilities of text-based large language models to the speech domain through just a single round of continuous speech pretraining on 30,000 hours of data—without any speech-specific instruction fine-tuning. SpeechCombine innovatively employs weight-difference fusion, integrating model weights before and after text-based instruction tuning to preserve original textual knowledge while endowing the model with robust spoken instruction understanding. This approach substantially reduces reliance on annotated speech instruction data and transcends limitations of conventional paradigms.

Abstract

Instruction tuning for speech language models (SLMs) is substantially more challenging than for text-based large language models (LLMs), as it requires learning a new modality and a wide range of speech-specific instructions in addition to those supported by text LLMs. Existing SLM training approaches largely replicate the text LLM training paradigm by synthesizing large-scale speech pre-training and instruction-tuning datasets. However, this strategy is difficult to scale, since speech sequences are significantly longer than text sequences. In this paper, we propose SpeechCombine, an instruction-following speech language model trained without any instruction tuning, using only a single round of speech pre-training on 30k hours of data. Starting from a text LLM base model, we perform continuous pre-training on speech utterances to obtain a speech-adapted model, and then directly combine its weights with the weight difference between the instruction-tuned and base versions of the text LLM. Our results show that this simple combination strategy not only preserves the knowledge and capabilities of the original text LLM, but also effectively transfers them to the speech domain. These findings suggest a new direction for SLM training that avoids reliance on massive speech data.

Problem

• speech language models
• instruction tuning
• modality adaptation
• speech-text compositionality
• pre-training

Innovation

• Speech Language Models
• Instruction Tuning
• Weight Combination
• Modality Transfer
• Pre-training