Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization

2025-03-10

AI Summary

This work investigates the learning dynamics of Low-Rank Adaptation (LoRA) for matrix factorization tasks, focusing on convergence behavior and initialization effects under gradient flow (GF). We theoretically analyze how initialization influences subspace alignment between the pretrained model and the target matrix. We show that small-scale initialization improves stability but exacerbates misalignment of singular subspaces, leading to a non-vanishing lower bound on convergence error. In contrast, spectral initialization precisely aligns the left and right singular subspaces of the pretrained model with those of the target matrix. We provide the first rigorous proof that, under the GF assumption, spectral initialization guarantees LoRA convergence to an arbitrarily accurate optimal low-rank solution. Empirical validation on matrix factorization and image classification tasks confirms that spectral initialization significantly outperforms standard initialization in reconstruction accuracy and generalization performance. Our results establish a new theoretical foundation for LoRA and introduce a principled initialization paradigm for practical deployment.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Despite the empirical success of Low-Rank Adaptation (LoRA) in fine-tuning pre-trained models, there is little theoretical understanding of how first-order methods with carefully crafted initialization adapt models to new tasks. In this work, we take the first step towards bridging this gap by theoretically analyzing the learning dynamics of LoRA for matrix factorization (MF) under gradient flow (GF), emphasizing the crucial role of initialization. For small initialization, we theoretically show that GF converges to a neighborhood of the optimal solution, with smaller initialization leading to lower final error. Our analysis shows that the final error is affected by the misalignment between the singular spaces of the pre-trained model and the target matrix, and reducing the initialization scale improves alignment. To address this misalignment, we propose a spectral initialization for LoRA in MF and theoretically prove that GF with small spectral initialization converges to the fine-tuning task with arbitrary precision. Numerical experiments from MF and image classification validate our findings.

Problem

• Theoretical understanding of LoRA learning dynamics in matrix factorization.
• Impact of initialization on convergence and final error in LoRA.
• Proposing spectral initialization to improve alignment and precision.

Innovation

• Analyzes LoRA dynamics via gradient flow perspective
• Proposes spectral initialization to reduce misalignment
• Validates findings with matrix factorization experiments