Normalized Low-Rank Adaptation

2026-08-31

AI Summary

为解决低秩适应（LoRA）训练动态不稳定的问题，提出归一化低秩适应（NoRA），通过在训练中或仅初始化时对下投影矩阵进行归一化处理，以提高收敛速度、性能和训练稳定性。

Abstract

While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.

Problem

• low-rank adaptation
• training dynamics
• optimization

Innovation

• Normalized Low-Rank Adaptation
• Down-projection Normalization
• Training Dynamics