On the Convergence Rate of LoRA Gradient Descent

2025-12-20

AI Summary

LoRA fine-tuning lacks theoretical convergence guarantees in finite iterations due to its absence of Lipschitz smoothness, relying instead on asymptotic analysis or strong boundedness assumptions. Method: This paper conducts the first non-asymptotic convergence analysis of the original LoRA gradient descent algorithm, eliminating all artificial assumptions—including Lipschitz continuity and parameter boundedness—by introducing an outer-product reparameterization technique and establishing a corrected descent lemma grounded in non-convex optimization theory. Contribution/Results: We rigorously prove that LoRA converges to a stationary point at a rate of $O(1/log T)$. This constitutes the first finite-step, assumption-free convergence guarantee for LoRA, uncovering its intrinsic convergence mechanism in practical training and substantially advancing the theoretical foundations of low-rank adaptation.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Learning Theory
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two ``adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the extit{original LoRA gradient descent} algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter matrices, ii) a modified descent lemma for the ``Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate $O(frac{1}{log T})$, where $T$ is the number of iterations.

Problem

• Analyzes convergence of original LoRA gradient descent algorithm
• Addresses lack of Lipschitz smoothness in theoretical analysis
• Provides non-asymptotic convergence rate without strong assumptions

Innovation

• Analyzes original LoRA gradient descent non-asymptotically
• Reformulates problem via outer product of stacked adapters
• Uses modified descent lemma for Lipschitz-like function