Cross-Modal Knowledge Distillation for Speech Large Language Models

2025-09-18

AI Summary

This work identifies and systematically evaluates two critical issues in speech large language models (Speech-LLMs): catastrophic forgetting—significant degradation of textual capabilities upon integrating speech modalities—and modality inequivalence—substantially lower performance on speech inputs compared to text inputs. To address these, we propose the first dual-channel cross-modal knowledge distillation framework, transferring knowledge from a pure-text teacher model to the Speech-LLM via two parallel pathways: text-to-text and speech-to-text. Our method integrates cross-modal alignment modeling with end-to-end joint training, supporting both dialogue and audio understanding tasks. Experiments demonstrate substantial improvements in speech-input performance across multiple benchmarks, while reducing textual capability degradation by up to 72%. This marks the first approach achieving synergistic optimization—enhancing speech understanding without compromising pre-existing text comprehension and reasoning abilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In this work, we present the first systematic evaluation of catastrophic forgetting and modality inequivalence in speech large language models, showing that introducing speech capabilities can degrade knowledge and reasoning even when inputs remain textual, and performance further decreases with spoken queries. To address these challenges, we propose a cross-modal knowledge distillation framework that leverages both text-to-text and speech-to-text channels to transfer knowledge from a text-based teacher model to a speech LLM. Extensive experiments on dialogue and audio understanding tasks validate the effectiveness of our approach in preserving textual knowledge, improving cross-modal alignment, and enhancing reasoning in speech-based interactions.

Problem

• Addressing catastrophic forgetting in speech LLMs
• Mitigating modality inequivalence in speech models
• Enhancing cross-modal knowledge and reasoning

Innovation

• Cross-modal knowledge distillation framework
• Text-to-text and speech-to-text channels
• Transfer knowledge from text teacher model