MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning

2026-08-15

AI Summary

This study addresses the negative transfer problem in heterogeneous federated instruction tuning caused by overlooking Mixture-of-Experts (MoE) routing behaviors. To this end, we propose ClientMorpher, a novel framework that leverages pretrained MoE routing signatures as a collaboration basis. By designing a dual-perspective clustering strategy encompassing both client and expert views, our method achieves routing-aware personalized federated learning. Experimental results demonstrate that, without increasing communication overhead, ClientMorpher effectively mitigates negative transfer induced by data heterogeneity. Furthermore, it significantly outperforms baselines such as FedAvg and local training in personalization performance, establishing a new paradigm for fine-tuning heterogeneous MoE models within federated settings.

Abstract

Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing. Recent Mixture-of-Experts (MoE) LLMs are particularly attractive for federated learning because their sparse activation reduces computation and communication while scaling model capacity. However, existing federated MoE methods primarily focus on parameter aggregation and personalization, overlooking the routing behavior of MoE models as a source of information for client collaboration. Under heterogeneous instruction distributions, indiscriminate aggregation can lead to negative transfer, highlighting the need to identify which clients should collaborate during federated optimization. We propose ClientMorpher, a routing-aware, personalized federated instruction fine-tuning framework that leverages routing signatures from pretrained MoE models to organize client collaboration prior to aggregation. We investigate two complementary clustering strategies: ClientMorpher-C, which directly clusters clients using expert activation profiles, and ClientMorpher-E, which first clusters experts based on their cross-client usage signatures and then derives client collaboration groups. We evaluate ClientMorpher for federated instruction fine-tuning on the Databricks Dolly-15K dataset, using pathological and Dirichlet-based heterogeneous client distributions across multiple instruction-following tasks. Experimental results show that routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging and local training, while maintaining the same communication cost. Furthermore, our study shows that client-centric and expert-centric clustering provides an effective and scalable approach for personalized federated instruction fine-tuning of sparse MoE LLMs.

Problem

• Federated Instruction Tuning
• Mixture-of-Experts
• Heterogeneous Data
• Negative Transfer
• Client Collaboration

Innovation

• Federated Instruction Tuning
• Mixture-of-Experts
• Routing-Aware Clustering
• Personalized Federated Learning
• Negative Transfer Mitigation