ElaLoRA: Elastic&Learnable Low-Rank Adaptation for Efficient Model Fine-Tuning

2025-03-31

AI Summary

Existing LoRA methods employ fixed or statically adjusted ranks, failing to account for inter-layer importance heterogeneity and resulting in inefficient parameter utilization. This work proposes the first adaptive LoRA framework that dynamically and elastically adjusts the rank of each layer during training. It introduces learnable, gradient-based importance scoring for layer-wise rank allocation, coupled with a differentiable rank parameterization and an elastic scheduling mechanism enabling concurrent rank pruning and expansion. The framework is fully compatible with the PEFT ecosystem. On multi-task benchmarks, it achieves an average accuracy gain of 1.8% under identical parameter budgets—outperforming state-of-the-art PEFT methods. Ablation studies confirm that layers assigned higher ranks contribute most critically to performance improvement.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-Rank Adaptation (LoRA) has become a widely adopted technique for fine-tuning large-scale pre-trained models with minimal parameter updates. However, existing methods rely on fixed ranks or focus solely on either rank pruning or expansion, failing to adapt ranks dynamically to match the importance of different layers during training. In this work, we propose ElaLoRA, an adaptive low-rank adaptation framework that dynamically prunes and expands ranks based on gradient-derived importance scores. To the best of our knowledge, ElaLoRA is the first method that enables both rank pruning and expansion during fine-tuning. Experiments across multiple benchmarks demonstrate that ElaLoRA consistently outperforms existing PEFT methods across different parameter budgets. Furthermore, our studies validate that layers receiving higher rank allocations contribute more significantly to model performance, providing theoretical justification for our adaptive strategy. By introducing a principled and adaptive rank allocation mechanism, ElaLoRA offers a scalable and efficient fine-tuning solution, particularly suited for resource-constrained environments.

Problem

• Dynamic rank adaptation for efficient model fine-tuning
• Combining rank pruning and expansion during training
• Optimizing layer-specific rank allocation for performance

Innovation

• Dynamic rank pruning and expansion
• Gradient-derived importance scores
• Adaptive low-rank adaptation framework