RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models

2025-05-24

AI Summary

LoRA fine-tuning suffers from imbalanced parameter updates and slow convergence due to the non-uniqueness of low-rank decomposition. To address this, we propose Stepwise Optimal Low-Rank Reconstruction (SOLR): at each training step, SOLR dynamically re-decomposes the low-rank adaptation matrices based on gradient analysis, optimizing toward minimization of a theoretically grounded loss upper bound. This yields balanced parameter updates and flattens the loss landscape. SOLR requires no architectural modifications and is plug-and-play compatible with mainstream LLMs (e.g., DeBERTaV3, LLaMA). On NLU and commonsense reasoning benchmarks, SOLR accelerates convergence by 22–35% over LoRA and QLoRA while significantly improving generalization; its computational overhead is negligible. Our key contribution is the first integration of real-time low-rank reconstruction with loss upper-bound-driven optimization into the adapter-based fine-tuning paradigm.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix. Albeit efficient, LoRA exhibits suboptimal convergence and noticeable performance degradation, due to inconsistent and imbalanced weight updates induced by its nonunique low-rank factorizations. To overcome these limitations, this article identifies the optimal low-rank factorization per step that minimizes an upper bound on the loss. The resultant refactored low-rank adaptation (RefLoRA) method promotes a flatter loss landscape, along with consistent and balanced weight updates, thus speeding up stable convergence. Extensive experiments evaluate RefLoRA on natural language understanding, and commonsense reasoning tasks with popular large language models including DeBERTaV3, LLaMA-7B, LLaMA2-7B and LLaMA3-8B. The numerical tests corroborate that RefLoRA converges faster, outperforms various benchmarks, and enjoys negligible computational overhead compared to state-of-the-art LoRA variants.

Problem

• Optimizes low-rank factorization for efficient large model fine-tuning
• Addresses inconsistent weight updates in LoRA to improve convergence
• Enhances performance with negligible computational overhead versus LoRA

Innovation

• Optimal low-rank factorization per step
• Flatter loss landscape promotion
• Negligible computational overhead