LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

2024-07-25

arXiv.org

AI Summary

LoRA achieves computational efficiency but suffers from substantial performance degradation compared to full-parameter fine-tuning. This work identifies the fundamental bottleneck as the insufficient approximation accuracy of low-rank gradients to the full gradient and establishes, for the first time, a rigorous gradient equivalence relationship between LoRA and full fine-tuning. Building upon this theoretical foundation, we propose LoRA-Pro: a method that designs a low-rank gradient calibration mechanism grounded in the theoretically optimal solution, employing a provably sound gradient reweighting strategy to enhance the representational capacity of low-rank updates. LoRA-Pro introduces no inference overhead—only learnable scaling factors are added. Extensive experiments across natural language understanding, dialogue generation, mathematical reasoning, code synthesis, and image classification demonstrate that LoRA-Pro significantly outperforms standard LoRA and closely approaches full fine-tuning performance. Our work provides both a novel theoretical perspective and a practical framework for parameter-efficient fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning of foundation models. Despite its computational efficiency, LoRA still yields inferior performance compared to full fine-tuning. In this paper, we first uncover a fundamental connection between the optimization processes of LoRA and full fine-tuning: using LoRA for optimization is mathematically equivalent to full fine-tuning using a low-rank gradient for parameter updates. And this low-rank gradient can be expressed in terms of the gradients of the two low-rank matrices in LoRA. Leveraging this insight, we introduce LoRA-Pro, a method that enhances LoRA's performance by strategically adjusting the gradients of these low-rank matrices. This adjustment allows the low-rank gradient to more accurately approximate the full fine-tuning gradient, thereby narrowing the performance gap between LoRA and full fine-tuning. Furthermore, we theoretically derive the optimal solutions for adjusting the gradients of the low-rank matrices, applying them during fine-tuning in LoRA-Pro. We conduct extensive experiments across natural language understanding, dialogue generation, mathematical reasoning, code generation, and image classification tasks, demonstrating that LoRA-Pro substantially improves LoRA's performance, effectively narrowing the gap with full fine-tuning. Code is publicly available at https://github.com/mrflogs/LoRA-Pro.

Problem

• Improving LoRA performance vs full fine-tuning
• Optimizing low-rank matrix gradient approximation
• Bridging performance gap in parameter-efficient tuning

Innovation

• LoRA-Pro adjusts low-rank matrix gradients strategically
• Mathematically links LoRA optimization to full fine-tuning
• Theoretically derives optimal gradient adjustment solutions