Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling

2026-04-01

AI Summary

This work addresses the significant degradation of textual capabilities often observed when adapting pretrained large language models (LLMs) to speech-language modeling via continued pretraining. To mitigate this trade-off, the authors propose a multimodal deep extension strategy that freezes the original LLM backbone and inserts a small number of additional Transformer layers, trained with an E-Branchformer architecture specifically designed for automatic speech recognition (ASR). Applied to SmolLM2-1.7B, this approach trains only approximately 40% of the model’s parameters yet achieves ASR performance on par with or superior to full fine-tuning, while reducing textual capability degradation by over 75%. The method thus effectively balances strong speech understanding with robust retention of original text-based competencies.

Abstract

Adapting pre-trained text Large Language Models (LLMs) into Speech Language Models (Speech LMs) via continual pretraining on speech data is promising, but often degrades the original text capabilities. We propose Multimodal Depth Upscaling, an extension of an emerging strategy in continual LLM pre-training, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data. Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning while causing far less text degradation than both full fine-tuning and Low-Rank Adaptation (LoRA). We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters.

Problem

• Speech Language Models
• Continual Pretraining
• Text Degradation
• Multimodal Adaptation
• Large Language Models

Innovation

• Multimodal Depth Upscaling
• Speech Language Models
• E-Branchformer
• Continual Pretraining
• Parameter-Efficient Adaptation