Expert Merging in Sparse Mixture of Experts with Nash Bargaining

2025-10-17

AI Summary

Sparse Mixture-of-Experts (SMoE) models lack principled, theoretically grounded weighting mechanisms for expert fusion. Method: This paper proposes a game-theoretic expert collaboration framework, wherein the Nash bargaining solution is employed to formally model cooperative and competitive interactions among experts, yielding an interpretable, dynamically adaptive weighting strategy; further integrated with a complex-domain momentum optimizer possessing guaranteed convergence, enhancing both parameter fusion efficiency and stability. Contribution/Results: The method is architecture-agnostic—compatible with mainstream MoE designs without modifying base model structures. Experiments demonstrate significant improvements over existing fusion approaches across language modeling, text classification, and robustness benchmarks. It has been successfully deployed on Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), achieving state-of-the-art performance in both zero-shot and fine-tuning settings.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Multiagent Systems: Mechanism Design

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings.

Problem

• Improving expert merging strategies in Sparse Mixture of Experts
• Establishing principled weighting mechanisms through game theory
• Enabling balanced collaboration among experts with theoretical guarantees

Innovation

• Introduces Nash Bargaining for balanced expert collaboration
• Incorporates complex momentum to accelerate expert propagation
• Seamlessly integrates with and scales large MoE architectures