Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition

2025-07-08

AI Summary

In speech recognition, sparse Mixture-of-Experts (MoE) models suffer from insufficient expert specialization due to layer-wise independent router decisions and lack of inter-layer coordination. To address this, we propose a **cross-layer shared router mechanism**—the first to reuse a single lightweight router across all Transformer layers in MoE architectures, enforcing routing consistency across depth and facilitating structured expert division of labor and collaboration. Our method leverages large-scale pseudo-labeled data for training. On ten mainstream ASR benchmarks, it achieves an average 11.2% relative WER reduction over dense Transformers and an 8.2% reduction over Switch Transformers, with faster convergence and lower training loss. The design significantly improves model robustness and parameter efficiency, establishing a scalable paradigm for inter-layer coordination in speech MoE architectures.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Speech

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Mixture-of-experts (MoE) architectures have expanded from language modeling to automatic speech recognition (ASR). Traditional MoE methods, such as the Switch Transformer, route experts independently within each layer. Our analysis reveals that routers in most layers make expert choices that are not strongly correlated with the choices of the routers in other layers. To increase the cooperation between experts in different layers and encourage greater specialization, we use a shared router across different MoE layers. We call this model emph{Omni-router Transformer}. Extensive experiments on a large-scale pseudo-labeled dataset and evaluations across 10 diverse, out-of-domain ASR benchmarks demonstrate that the Omni-router Transformer is able to achieve lower training loss and consistently outperform dense and Switch Transformer models, reducing average word error rates by 11.2% and 8.2%, respectively, while providing structured expert usage and improved robustness to diverse data.

Problem

• Improving expert cooperation in Mixture-of-Experts for ASR
• Reducing word error rates in speech recognition models
• Enhancing robustness and specialization in sparse MoE architectures

Innovation

• Shared router across MoE layers
• Enhances expert cooperation and specialization
• Reduces word error rates significantly