🤖 AI Summary
This work addresses the limitations of existing LoRA fine-tuning theory, which imposes overly stringent rank requirements and lacks adaptation to cross-entropy loss in binary classification settings. By refining manifold dimension analysis and incorporating the Polyak–Łojasiewicz inequality with Rademacher complexity bounds, the authors establish, for the first time within the neural tangent kernel (NTK) framework, optimization guarantees for binary classification tasks that require either no rank threshold or an extremely low rank—such as rank 1. Theoretical analysis demonstrates that the necessary LoRA rank can be reduced from the commonly recommended value of 12 down to 1. Empirical validation across four GLUE binary classification benchmarks and multiple encoder architectures confirms that rank-1 LoRA achieves performance comparable to rank-12, while multi-class tasks exhibit higher optimal ranks, aligning with theoretical predictions.
📝 Abstract
A recent landscape analysis of LoRA fine-tuning in the neural tangent kernel regime establishes a sufficient condition $r(r+1)/2 > KN$ on the LoRA rank $r$ for the absence of spurious local minima under squared-error loss, prescribing $r \geq 12$ on canonical few-shot RoBERTa setups. The condition is stated for general output dimension $K$, so its sharpness in any particular regime, and its practical implication for the cross-entropy loss actually used in fine-tuning, are open. We give three results that together reduce the prescribed rank to $r = 1$ for binary classification in this regime. First, replacing the symmetric Sard-form count with the non-symmetric LoRA manifold dimension yields a strictly weaker capacity requirement, $r(m+n) - r^2 > C^* \cdot KN$ with $C^* \approx 1.35$ under Gaussian-iid features, satisfied at $r = 1$ on canonical setups. Second, in the cross-entropy setting the Polyak--Łojasiewicz inequality removes the rank threshold entirely. Third, a Rademacher-complexity bound predicts rank-one variance optimality precisely when the bias term is saturated, which is the case for binary classification but not for $K > 2$. Empirically, across four GLUE-style binary tasks, three encoder architectures, and at scale on RoBERTa-large, rank one is competitive with the existing prescription $r = 12$; on multi-class MNLI the optimal rank shifts above one, also as predicted. The binary-regime guarantees are conditional on standard NTK assumptions; the multi-class extension is left to future work.