Unifying Model and Layer Fusion for Speech Foundation Models

2025-11-11

AI Summary

To address the low representation fusion efficiency of speech foundation models in downstream tasks, this paper proposes the first unified framework integrating both model-level and layer-level fusion. The core innovation is a lightweight cross-model interface module that enables joint alignment and end-to-end optimization across multiple upstream speech models—whether self-supervised or supervised—as well as their internal layer-wise features. This module facilitates deep semantic integration across heterogeneous models, significantly enhancing fusion flexibility and representational capacity. Evaluated on automatic speech recognition (ASR) and language-like analysis tasks, the method consistently outperforms existing fusion strategies. It demonstrates strong scalability and consistent performance gains across varying model scales and numbers of constituent models, establishing a new state-of-the-art in speech representation fusion.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Speech Foundation Models have gained significant attention recently. Prior works have shown that the fusion of representations from multiple layers of the same model or the fusion of multiple models can improve performance on downstream tasks. We unify these two fusion strategies by proposing an interface module that enables fusion across multiple upstream speech models while integrating information across their layers. We conduct extensive experiments on different self-supervised and supervised models across various speech tasks, including ASR and paralinguistic analysis, and demonstrate that our method outperforms prior fusion approaches. We further analyze its scalability concerning model size and count, highlighting the importance of selecting appropriate upstream models. Our results show that the proposed interface provides an additional performance boost when given a suitable upstream model selection, making it a promising approach for utilizing Speech Foundation Models.

Problem

• Unifying model and layer fusion strategies for speech foundation models
• Improving performance across speech tasks like ASR and paralinguistic analysis
• Enhancing scalability through optimal upstream model selection and integration

Innovation

• Unified fusion across models and layers
• Interface module integrates multiple upstream models
• Scalable approach for speech foundation models