Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving

2026-07-02

AI Summary

This work addresses the limitations of existing speech–large language models (Speech-LLMs) in effectively integrating textual priors for automatic speech recognition (ASR), particularly their diminishing advantage under large-scale labeled data and insufficient utilization of text in conventional joint training paradigms. To overcome these challenges, the authors propose a Joint Speech–Text Interleaved Pretraining strategy (JSTIP) tailored for ASR. By constructing word-level and segment-level interleaved speech–text sequences and introducing a novel modality-interleaved input architecture, JSTIP enables efficient adaptation using only in-domain text—without requiring synthetic speech—while preserving the generative priors of the underlying LLM. Evaluated on a Speech-LLM architecture supporting continuous inputs and trained on 38k hours of ASR data, the method achieves state-of-the-art performance among open-source systems in named entity recognition accuracy and demonstrates emergent zero-shot spoken question answering capabilities.

Abstract

Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recognition (ASR) remain unclear. We observe that as supervised ASR training data increases, the contribution of LLM priors becomes less evident, and simple speech-text joint training under-utilizes textual knowledge. We therefore propose Joint Speech-Text Interleaved Pretraining (JSTIP), an ASR-oriented pretraining strategy that constructs word-level and segment-level interleaved speech-text sequences within aligned pairs for speech-LLM architectures that accept continuous inputs. Experiments on 38k hours of ASR data show consistent entity accuracy improvement compared to ASR-only and joint speech-text training baselines. JSTIP achieves on-par entity recognition performance using domain transcription text compared to synthetic speech-text pairs, simplifying domain adaptation. Benefiting from textual pretraining and domain text data, JSTIP is competitive with open-source ASR and Speech-LLM systems in medical entity recognition. The zero-shot speech question answering behaviors further suggest that interleaving reduces the speech-text modality gap and preserves the LLM generative prior, which is likely the reason for the entity improvements on the ASR task.

Problem

• Speech-LLM integration
• automatic speech recognition
• textual pretraining
• joint training
• modality gap

Innovation

• Speech-LLM integration
• interleaved pretraining
• joint speech-text training
• entity recognition
• modality gap