Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts

2026-06-14

AI Summary

In heterogeneous federated learning, Mixture-of-Experts (MoE) models suffer from aggregation interference due to conflicting optimization objectives across clients, which impedes convergence and performance. This work proposes FC-MoE, the first framework to systematically address expert conflict in federated MoE settings. FC-MoE employs importance-aware weighting to filter reliable local updates, integrates gradient consensus projection to suppress conflicting update directions, and introduces a local knowledge retention mechanism that anchors domain-specific residuals to stabilize the global optimization trajectory. The proposed approach significantly accelerates convergence under non-IID data distributions while simultaneously enhancing the performance of both the global model and individual client models.

Abstract

The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixture-of-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation. While federated learning (FL) emerges as the paradigm for privacy-preserving collaborative optimization, integrating MoE into FL under data heterogeneity may trigger conflicting expert optimizations. Client-specific data distributions force same-indexed experts to optimize under inconsistent or even conflicting feature-label correlations. This mismatch induces destructive interference during aggregation, thus destabilizing the optimization trajectory and degrading model performance. To address this issue, we propose FC-MoE, a federated conflict-aware framework for MoE fine-tuning. It employs an importance aware weighting scheme to prioritize reliable local updates and utilizes gradient consensus projection to suppress conflicting updates, ensuring a stable global optimization path. Moreover, a local knowledge retention mechanism further preserves specialized client expertise by re-anchoring domain-specific residuals. Extensive experiments demonstrate that FC-MoE accelerates convergence and enhances both global and local model performance in non-IID federated environments.

Problem

• Federated Learning
• Mixture-of-Experts
• Data Heterogeneity
• Expert Conflict
• Large Language Models

Innovation

• Federated Learning
• Mixture-of-Experts
• Conflict-Aware Optimization
• Gradient Consensus Projection
• Non-IID