AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert

2025-11-23

AI Summary

Existing multimodal Mixture-of-Experts (MoE) models employ fixed expert activation strategies, overlooking the heterogeneity in semantic importance across modalities and tokens, leading to inefficient resource allocation. This paper proposes AnyExperts—a budget-aware, on-demand dynamic routing framework. It adaptively determines the number of experts activated per token based on token-level semantic importance estimation. To enable fine-grained, importance-driven computation allocation under a fixed total slot budget, we introduce a novel virtual expert mechanism that dynamically balances the ratio of real to virtual experts. The method is modality-agnostic, supporting vision, audio, and text. Under identical FLOPs budgets, AnyExperts reduces real expert activation by 40% for image/video tasks and by 10% for text tasks—without sacrificing accuracy—thereby significantly improving both efficiency and performance.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal Mixture-of-Experts (MoE) models offer a promising path toward scalable and efficient large vision-language systems. However, existing approaches rely on rigid routing strategies (typically activating a fixed number of experts per token) ignoring the inherent heterogeneity in semantic importance across modalities. This leads to suboptimal compute allocation, where redundant tokens consume as many resources as critical ones. To address this, we propose AnyExperts, a novel on-demand, budget-aware dynamic routing framework that allocates a variable total number of expert slots per token based on its semantic importance. Crucially, to prevent uncontrolled compute growth, the total slots per token are constrained within a fixed range, and each slot is filled by either a real expert or a virtual expert, with the virtual share capped at a small maximum (e.g., 20%). The model then adaptively balances the real-to-virtual ratio per token, assigning more real experts to semantically rich regions and relying more on virtual experts for redundant content. Evaluated across diverse tasks in visual understanding, audio understanding, and NLP understanding, AnyExperts improves performance under the same compute budget. Notably, on general image/video tasks, it achieves comparable accuracy with 40% fewer real expert activations; on text-dense tasks (OCR and NLP), it maintains performance while reducing real expert usage by 10%. These results demonstrate that fine-grained, importance-driven expert allocation significantly enhances both the efficiency and effectiveness of multimodal MoE models.

Problem

• Addresses suboptimal compute allocation in multimodal MoE models
• Proposes dynamic routing for variable expert slots per token
• Balances real and virtual experts to enhance efficiency

Innovation

• Dynamic routing framework allocates expert slots per token
• Balances real and virtual experts based on semantic importance
• Constrains total slots per token within fixed compute budget