Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning

2026-02-05

AI Summary

This work addresses the challenge that in LoRA fine-tuning, the optimal learning rate must be repeatedly tuned for different adapter ranks due to the absence of a unified scaling law and poor transferability to full-parameter fine-tuning. To resolve this, the authors propose μA, a theoretical framework grounded in Maximal-Update Parametrization (μP), which for the first time reveals two distinct optimal scaling laws governing the relationship between LoRA learning rates and adapter rank. Furthermore, μA establishes a principled mechanism for transferring learning rates from low-rank adaptation to full fine-tuning. Extensive experiments across language, vision, multimodal, image generation, and reinforcement learning tasks demonstrate that μA significantly reduces hyperparameter tuning costs and enables learning rates optimized for LoRA to be directly applied to full fine-tuning, thereby substantially improving training efficiency.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Low-Rank Adaptation (LoRA) is a standard tool for parameter-efficient finetuning of large models. While it induces a small memory footprint, its training dynamics can be surprisingly complex as they depend on several hyperparameters such as initialization, adapter rank, and learning rate. In particular, it is unclear how the optimal learning rate scales with adapter rank, which forces practitioners to re-tune the learning rate whenever the rank is changed. In this paper, we introduce Maximal-Update Adaptation ($\mu$A), a theoretical framework that characterizes how the"optimal"learning rate should scale with model width and adapter rank to produce stable, non-vanishing feature updates under standard configurations. $\mu$A is inspired from the Maximal-Update Parametrization ($\mu$P) in pretraining. Our analysis leverages techniques from hyperparameter transfer and reveals that the optimal learning rate exhibits different scaling patterns depending on initialization and LoRA scaling factor. Specifically, we identify two regimes: one where the optimal learning rate remains roughly invariant across ranks, and another where it scales inversely with rank. We further identify a configuration that allows learning rate transfer from LoRA to full finetuning, drastically reducing the cost of learning rate tuning for full finetuning. Experiments across language, vision, vision--language, image generation, and reinforcement learning tasks validate our scaling rules and show that learning rates tuned on LoRA transfer reliably to full finetuning.

Problem

• LoRA
• learning rate scaling
• adapter rank
• full finetuning
• hyperparameter transfer

Innovation

• LoRA
• learning rate scaling
• parameter-efficient finetuning
• hyperparameter transfer
• Maximal-Update Adaptation