GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Models

2026-06-24

AI Summary

This work addresses the inefficiency of traditional Mixture-of-Experts (MoE) architectures, which rely on static Top-k routing and cannot dynamically adjust the number of activated experts per input. Framing token routing as an information encoding task, the study establishes a novel connection between gating entropy and the Minimum Description Length (MDL) principle, leading to an uncertainty-aware adaptive routing mechanism that explicitly balances model complexity against performance. Evaluated across multiple backbone networks and benchmarks, the proposed method achieves a 36.5% improvement in expert activation sparsity while maintaining 99.5% of the original static routing performance, significantly outperforming existing static and heuristic dynamic routing strategies.

Abstract

With the increase in model parameters and training data, the instruction following and generalization capabilities of Large VisionLanguage Models (LVLMs) have been significantly improved. Based on the Mixture of Experts (MoE) architecture, LVLMs expand their parameter capacity while maintaining the inference cost. However, traditional MoE methods employ a Top-k static routing strategy, which fails to account for variations in the input and adaptively select the number of experts, resulting in suboptimal resource utilization. In this paper, we propose viewing token routing as an information encoding task, framing dynamic routing as a Minimum Description Length (MDL) problem in encoding By validating the connection between MDL and gating entropy in the MoE scenario, we introduce Gating Entropy-based Uncertainty-aware Adaptive Routing (GeMoE) for MoE. Unlike traditional static or heuristic-based dynamic routing methods, GeMoE explicitly models the trade-off between model complexity and performance. By using gating entropy to assess the complexity of tokens, GeMoE adaptively determines the number of experts each token should engage. On a wide range of backbones and benchmarks, our method achieves 99.5% average performance retention compared to the original static routing, while improving average expert activation sparsity by 36.5%.

Problem

• Mixture of Experts
• adaptive routing
• static routing
• expert selection
• resource utilization

Innovation

• Mixture of Experts
• Gating Entropy
• Minimum Description Length
• Adaptive Routing
• Vision-Language Models