SSR: Alignment-Aware Modality Connector for Speech Language Models

2024-09-30

arXiv.org

AI Summary

This work addresses two key challenges in speech-language model fusion: inefficient long-speech encoding and catastrophic forgetting of textual capabilities. We propose the Speech-Text Alignment-driven Segmented Speech Representation Connector (SSR-Connector). Its core contributions are: (1) an alignment-aware speech segmentation and compression mechanism, which leverages forced speech-text alignment to achieve efficient, semantically preserved dimensionality reduction of speech features; and (2) a distillation-guided two-stage incremental training paradigm that injects speech understanding capability while robustly preserving the pre-trained language model’s textual competence. Evaluated on StoryCloze and Speech-MMLU benchmarks, SSR-Connector achieves +10% and +20% absolute accuracy gains over existing fusion approaches, respectively, with zero degradation in original text-only performance.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-Connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-Connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., +10 accuracy on StoryCloze and +20 on Speech-MMLU) while preserving pre-trained text ability.

Problem

• Inefficient encoding of long-form speech in SpeechLM
• Catastrophic forgetting of pre-trained text modality
• Improving speech-text modality fusion performance

Innovation

• Segments and compresses speech features
• Uses two-stage training pipeline
• Leverages speech-text alignments