FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models

2026-02-09

AI Summary

This work addresses the parameter redundancy and efficiency–performance trade-off in traditional Mixture-of-Experts (MoE) architectures within federated large language models, where full-sized experts lead to excessive computational overhead. To overcome this limitation, the authors propose FlexMoRE, the first MoE framework incorporating rank-heterogeneous experts by constructing each expert as a low-rank adapter with varying ranks spanning from \(2^0\) to \(2^{14}\). Within a federated training paradigm, the study systematically investigates the interplay between expert rank and task performance. Empirical results reveal a marked divergence in sensitivity to expert rank between reasoning-intensive and knowledge-intensive tasks. Evaluated on 120 tasks across 150 configurations using the FlexOlmo framework, FlexMoRE achieves an average score of 47.18 with only 10.75B parameters—less than one-third of the baseline—outperforming full-sized MoE (45.46).

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in mixture-of-experts architectures have shown that individual experts models can be trained federatedly, i.e., in isolation from other experts by using a common base model to facilitate coordination. However, we hypothesize that full-sized experts may not be necessary for all domains and that instead low-rank adapters may be sufficient. Here, we introduce FlexMoRE, a Flexible Mixture of Rank-heterogenous Experts, which may be either full-sized experts or adapters of a suitable rank. We systematically investigate the trade-off between expert rank and downstream task performance by evaluating $6$ experts with ranks $2^0$ to $2^{14}$ resulting in experiments covering 150 mixtures (96 with 2 experts, 54 with 7 experts) that are evaluated across $120$ tasks. For our experiments, we build on FlexOlmo and turn its pre-trained experts into low-rank versions. Our regression analysis from expert rank to downstream task performance reveals that the best-performing rank is substantially higher for reasoning-heavy benchmarks than for knowledge-heavy benchmarks. These findings on rank sensitivity come with direct implications for memory efficiency: Using optimal ranks, FlexMoRE yields improved downstream task performance (average score $47.18$) compared to the baseline FlexOlmo-style mixture of full-sized experts (average score $45.46$) at less than one third the parameters ($10.75$B for FlexMoRE vs. $33.27$B for FlexOlmo). All code will be made available.

Problem

• mixture-of-experts
• federated training
• low-rank adaptation
• parameter efficiency
• large language models

Innovation

• Mixture of Experts
• Low-rank Adaptation
• Federated Learning
• Rank Heterogeneity
• Parameter Efficiency