Interleaved Speech Language Models Latently Work In Text

2026-06-21

AI Summary

This study addresses the unclear interaction mechanism between speech and text modalities in the latent space of existing speech-text interleaved language models. The work reveals, for the first time, that such models implicitly generate high-confidence textual transcriptions in intermediate layers—even without explicit speech recognition training—and subsequently perform language modeling based on these transcriptions. Leveraging the logit lens technique alongside ablation studies, the authors systematically analyze models of varying scales and architectures, highlighting the critical roles of interleaved data and text-based language model initialization. Experiments demonstrate that, in up to 77% of samples, the ground-truth transcript tokens appear among the top predictions at intermediate layers, confirming the presence of an implicit transcription stage and establishing its strong correlation with the model’s spoken language understanding capabilities.

Abstract

Speech language models (SLMs) have been extensively studied, with the common paradigm incorporating text data and pre-trained text LMs. A leading approach is speech-text interleaving in which models are trained over sequences containing both speech and text tokens, aiming to boost even speech-only capabilities. Yet the way these two modalities interact in the model latent space remains unclear. In this work, we analyze interleaved speech-text LMs from different model families and sizes through the scope of the logit lens to provide such insight. We reveal that these models go through an implicit transcription phase in which the text token of the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. The transcription of the word appears as one of the top candidate words for as much as 77\% of the data. Following this stage, the models proceed to predict the next word in the text space before transforming back to the speech domain. We finally analyze the role of interleaving data, and initializing from text LMs in eliciting this behavior, as well as seeing how this correlates with spoken knowledge abilities. Our analysis sheds light on the internal mechanisms underlying the relationship between speech and text modalities and could shape SLM optimization.

Problem

• speech language models
• interleaved speech-text
• latent space
• modality interaction
• logit lens

Innovation

• interleaved speech-text modeling
• logit lens analysis
• implicit transcription
• speech-language models
• modality interaction