TaRA: Training-Aware Low-Rank Adaptation Initialization

2026-09-02

AI Summary

本文提出TaRA方法，通过使低秩因子诱导的梯度逼近全秩权重矩阵的梯度，解决LoRA初始化敏感问题，提高训练初期梯度保真度。

Abstract

Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.

Problem

• Low-Rank Adaptation
• Initialization
• Training Dynamics

Innovation

• Training-aware
• Low-Rank Adaptation
• Gradient Fidelity
• Initialization
• Parameter-Efficient Fine-Tuning