MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts

2026-01-15

AI Summary

This work addresses the limitation of existing multimodal large language models that employ shared parameters for speech and text, thereby neglecting their inherent representational differences and impairing modality-specific learning. To overcome this, the authors propose MoST, a novel modality-aware mixture-of-experts (MAMoE) architecture that assigns dedicated expert pathways for speech and text while incorporating shared experts to facilitate cross-modal fusion. As the first fully open-source speech-text mixture-of-experts large language model, MoST integrates modality-aware routing, post-training on open-source data, speech-text instruction tuning, and efficient alignment techniques. It achieves significant performance gains over comparable-scale models across diverse tasks—including automatic speech recognition, text-to-speech synthesis, audio language modeling, and spoken question answering. Ablation studies further confirm the effectiveness of both modality-specific routing and the shared expert mechanism.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. \footnote{We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST

Problem

• multimodal large language model
• speech-text integration
• modality-specific representation
• Mixture of Experts
• cross-modal understanding

Innovation

• Mixture of Experts
• Multimodal LLM
• Modality-Aware Routing
• Speech-Text Integration
• Open-Source Training