CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

2026-06-30

AI Summary

Existing SVD-based low-rank adaptation methods overlook the coupled geometric structure between the left and right singular bases of pre-trained weights, limiting their stability and parameter efficiency. This work proposes a slice-wise consistent orthogonal rotation mechanism that, after SVD truncation, applies a shared orthogonal transformation and diagonal spectral shift to the left and right singular bases across all row slices. This approach explicitly models their geometric coupling while inducing minimal perturbation to the original weights. Notably, it achieves, for the first time, slice-level consistent orthogonal transformations, substantially reducing the number of trainable parameters. Experimental results demonstrate that the proposed method outperforms state-of-the-art approaches—including LoRA, DoRA, PiSSA, and MiLoRA—on commonsense reasoning and code generation tasks, using only approximately one-eighth of their trainable parameters.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) commonly adapts pretrained weights through low-rank updates, and recent methods further exploit the singular value decomposition (SVD) of the base weight for initialization or subspace selection. However, these methods do not explicitly preserve the coupled geometry between the pretrained left and right singular bases. Motivated by recent minimum-perturbation theory, which shows that stable finetuning follows a coherent SVD rotation in which a single orthogonal $Q$ acts on both the left singular basis $U_0$ and the right singular basis $V_0$, we prove a per-slice analogue: each row slice of $W_0$ can be adapted by a shared orthogonal rotation $Q_i$ on its left basis $U_i$ and right basis $V_i$ together with a diagonal spectrum shift. We implement this form as CORA (Coherent Orthogonal Rotation Adaptation), which applies per-slice orthogonal rotations and a per-layer diagonal scale to the rank-$r$ SVD truncation of $W_0$. CORA uses $\tfrac{1}{2}m(r{-}1)$ trainable parameters per linear layer, about $4{\times}$ fewer than LoRA at the same rank. CORA outperforms LoRA, DoRA, PiSSA, and MiLoRA on commonsense reasoning and code generation while using about $8{\times}$ fewer parameters.

Problem

• Parameter-Efficient Fine-Tuning
• Singular Value Decomposition
• Low-Rank Adaptation
• Orthogonal Rotation
• Pretrained Model Adaptation

Innovation

• Coherent Orthogonal Rotation
• Per-Slice Adaptation
• SVD-based Low-Rank Adaptation
• Parameter-Efficient Fine-Tuning
• Minimum-Perturbation Theory