ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning

2025-10-27

AI Summary

To address the trade-off between efficiency and performance in low-rank adaptation (LoRA) for large language model fine-tuning, this paper proposes Continuous Low-rank Incremental Accumulation (CLIC). CLIC introduces, for the first time, an analytical column-wise scaling mechanism that dynamically approximates the optimal low-rank update direction without restarting optimization, thereby unifying high-rank expressivity with low computational overhead. The method integrates low-rank decomposition, gradient direction analysis, and analytically derived optimal scaling to enable parameter-efficient, cumulative updates. Experiments on a 12B-parameter model demonstrate that CLIC significantly outperforms existing LoRA variants across natural language understanding, commonsense reasoning, and mathematical reasoning tasks—achieving up to 35% faster convergence and final performance markedly closer to full fine-tuning.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Learning Preferences or Rankings
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As large language models (LLMs) continue to scale in size, the computational overhead has become a major bottleneck for task-specific fine-tuning. While low-rank adaptation (LoRA) effectively curtails this cost by confining the weight updates to a low-dimensional subspace, such a restriction can hinder effectiveness and slow convergence. This contribution deals with these limitations by accumulating progressively a high-rank weight update from consecutive low-rank increments. Specifically, the per update optimal low-rank matrix is identified to minimize the loss function and closely approximate full fine-tuning. To endow efficient and seamless optimization without restarting, this optimal choice is formed by appropriately scaling the columns of the original low-rank matrix. Rigorous performance guarantees reveal that the optimal scaling can be found analytically. Extensive numerical tests with popular LLMs scaling up to 12 billion parameters demonstrate a consistent performance gain and fast convergence relative to state-of-the-art LoRA variants on diverse tasks including natural language understanding, commonsense reasoning, and mathematical problem solving.

Problem

• Overcoming low-rank adaptation limitations for efficient high-rank fine-tuning
• Optimally scaling low-rank increments to approximate full fine-tuning performance
• Enhancing convergence speed and effectiveness in large language model adaptation

Innovation

• Accumulates high-rank updates from low-rank increments
• Optimally scales low-rank columns for fine-tuning
• Analytically determines scaling for performance guarantees