DR-LoRA: Dynamic Rank LoRA for Mixture-of-Experts Adaptation

2026-01-08

arXiv.org

AI Summary

This work addresses the inefficiency of uniform LoRA rank allocation in parameter-efficient fine-tuning of Mixture-of-Experts (MoE) models, which overlooks functional heterogeneity among experts. To remedy this, the authors propose a dynamic rank allocation framework that adaptively assigns heterogeneous LoRA ranks during fine-tuning based on an expert salience scoring mechanism. This score integrates routing frequency and rank importance to identify high-impact experts, whose LoRA ranks are selectively expanded under a fixed parameter budget. Evaluated within the MoE architecture, the proposed method consistently outperforms standard LoRA and static rank allocation strategies, achieving superior task performance while enhancing parameter utilization efficiency.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Mixture-of-Experts (MoE) has become a prominent paradigm for scaling Large Language Models (LLMs). Parameter-efficient fine-tuning (PEFT), such as LoRA, is widely adopted to adapt pretrained MoE LLMs to downstream tasks. However, existing approaches assign identical LoRA ranks to all experts, overlooking the intrinsic functional specialization within MoE LLMs. This uniform allocation leads to resource mismatch, task-relevant experts are under-provisioned while less relevant ones receive redundant parameters. We propose a Dynamic Rank LoRA framework named DR-LoRA, which dynamically grows expert LoRA ranks during fine-tuning based on task-specific demands. DR-LoRA employs an Expert Saliency Scoring mechanism that integrates expert routing frequency and LoRA rank importance to quantify each expert's demand for additional capacity. Experts with higher saliency scores are prioritized for rank expansion, enabling the automatic formation of a heterogeneous rank distribution tailored to the target task. Experiments on multiple benchmarks demonstrate that DR-LoRA consistently outperforms standard LoRA and static allocation strategies under the same parameter budget, achieving superior task performance with more efficient parameter utilization.

Problem

• Mixture-of-Experts
• LoRA
• parameter-efficient fine-tuning
• rank allocation
• expert specialization

Innovation

• Dynamic Rank Allocation
• Mixture-of-Experts
• Parameter-Efficient Fine-Tuning
• LoRA
• Expert Saliency Scoring