🤖 AI Summary
To address the excessive number of trainable parameters in large language model (LLM) fine-tuning, this paper proposes QR-LoRA, a low-rank adaptation method based on column-pivoted QR decomposition. Unlike standard LoRA or SVD-initialized variants, QR-LoRA efficiently constructs an orthogonal basis for pretrained weights via QR decomposition and represents adaptation updates as sparse linear combinations within this basis—optimizing only a small set of coefficients. This design jointly enhances computational efficiency and interpretability. QR-LoRA is the first to integrate a structured orthogonal basis into the low-rank adaptation framework. On the GLUE benchmark, it matches or surpasses full fine-tuning, standard LoRA, and SVD-LoRA in performance. With as few as 601 trainable parameters, it reduces parameter count by over 1,000× compared to full fine-tuning and by 77× relative to typical LoRA, achieving substantial gains in parameter efficiency.
📝 Abstract
The growing scale of Large Language Models (LLMs) has necessitated the development of parameter-efficient fine-tuning techniques. Low-Rank Adaptation (LoRA) has emerged as a promising approach, reducing the number of trainable parameters by applying low-rank updates to pretrained weights. While standard LoRA learns both update factors directly, several recent variants first initialize those matrices via an SVD of the pretrained weights -- an operation that can be expensive on large models and yields singular vectors that are not always easy to interpret. In this work, we extract an orthonormal basis from the pretrained weight matrix using QR decomposition with column pivoting, and then express the LoRA update as a linear combination of these basis vectors -- training only the scalar coefficients, which imposes clear structure on adaptation and drastically reduces parameter count. Experiments across GLUE tasks show that QR-LoRA matches or exceeds the performance of full fine-tuning, standard LoRA, and SVD-LoRA (LoRA with update matrices initialized via singular value decomposition) with as few as 601 parameters -- a reduction of over 1000x compared to full fine-tuning and 77x fewer than typical LoRA setups.