Mixture of Heterogeneous Grouped Experts for Language Modeling

2026-04-24

AI Summary

Standard Mixture-of-Experts (MoE) models enforce homogeneous experts, limiting flexible allocation of computational resources according to token complexity. Existing heterogeneous MoE approaches suffer from imbalanced GPU loads and inefficient parameter utilization. To address these issues, this work proposes a Heterogeneous Mixture of Grouped Experts (MoHGE), which employs a two-level routing mechanism for resource-aware expert composition and introduces a group-level auxiliary loss to dynamically guide token assignment. A full-scale group decoupling strategy further ensures balanced computation across GPUs. By supporting expert heterogeneity while significantly improving parameter efficiency and hardware utilization, MoHGE reduces total model parameters by approximately 20% with comparable performance, thereby lowering inference costs and overcoming key deployment bottlenecks of heterogeneous MoE in industrial settings.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) based on Mixture-of-Experts (MoE) are pivotal in industrial applications for their ability to scale performance efficiently. However, standard MoEs enforce uniform expert sizes,creating a rigidity that fails to align computational costs with varying token-level complexity. While heterogeneous expert architectures attempt to address this by diversifying expert sizes, they often suffer from significant system-level challenges, specifically unbalanced GPU utilization and inefficient parameter utilization, which hinder practical deployment. To bridge the gap between theoretical heterogeneity and robust industrial application, we propose Mixture of Heterogeneous Grouped Experts (MoHGE) which introduces a two-level routing mechanism to enable flexible, resource-aware expert combinations. To optimize inference efficiency, we propose a Group-Wise Auxiliary Loss, which dynamically steers tokens to the most parameter-efficient expert groups based on task difficulty. To address the critical deployment challenge of GPU load balancing, we introduce an All-size Group-decoupling Allocation strategy coupled with an Intra-Group Experts Auxiliary Loss. These mechanisms collectively ensure uniform computation distribution across GPUs. Extensive evaluations demonstrate that MoHGE matches the performance of MoE architectures while reducing the total parameters by approximately 20% and maintaining balanced GPU utilization. Our work establishes a scalable paradigm for resource-efficient MoE design, offering a practical solution for optimizing inference costs in real-world scenarios.

Problem

• Mixture-of-Experts
• heterogeneous experts
• GPU load balancing
• parameter efficiency
• language modeling

Innovation

• Mixture-of-Experts
• heterogeneous experts
• GPU load balancing
• resource-efficient inference
• two-level routing