model adaptation and transfer learning

This work addresses the lack of theoretical understanding regarding the generalization gap between Low-Rank Adaptation (LoRA) and full fine-tuning. Within a simplified linear regression framework, the study systematically compares their generalization behaviors by integrating statistical learning theory with excess risk analysis. It establishes, for the first time, a theoretical guarantee that when the discrepancy between the pre-trained model and the downstream task exhibits a low-rank structure, LoRA achieves lower excess risk than full fine-tuning in both over-parameterized and under-parameterized regimes. Empirical experiments further corroborate this finding, demonstrating a non-intuitive improvement in test accuracy under low-rank constraints and thereby revealing the intrinsic mechanism underlying LoRA’s superior generalization capability.

To address the insufficient generalization of pretrained models when fine-tuned on few-shot, high-dimensional binary classification tasks, this paper proposes a novel fine-tuning framework based on weight matrix reparameterization. The method couples low-rank adaptation (LoRA) with a base-model rescaling mechanism and employs random matrix theory to model the generalization behavior of high-dimensional classifiers, thereby revealing how rescaling governs spectral distribution and generalization bounds. Theoretically, the approach substantially mitigates overfitting by controlling the effective rank and condition number of the classifier’s weight matrix. Empirically, it consistently improves performance across multiple binary classification benchmarks and large language model (LLM) fine-tuning tasks, demonstrating particularly pronounced generalization gains under extreme data scarcity.

To address catastrophic forgetting, degraded out-of-distribution (OOD) generalization, and high computational overhead in large-model domain adaptation, this paper proposes a parameter-efficient fine-tuning method based on selective activation of LoRA modules. Our core innovation is a learnable binary gating function that enables fine-grained, task-aware sparsity in LoRA updates, integrated within the Task Adaptive Parameter Sharing (TAPS) framework and low-rank decomposition. The method updates only ~5% of parameters. Evaluated on CLIP and DINO-ViT, it reduces trainable parameters by over 95% compared to standard LoRA, maintains or improves OOD accuracy, and significantly mitigates forgetting of prior-task knowledge. To our knowledge, this is the first work within the parameter-efficient fine-tuning (PEFT) paradigm to systematically enhance both OOD robustness and long-term knowledge retention.

LoRA achieves computational efficiency but suffers from substantial performance degradation compared to full-parameter fine-tuning. This work identifies the fundamental bottleneck as the insufficient approximation accuracy of low-rank gradients to the full gradient and establishes, for the first time, a rigorous gradient equivalence relationship between LoRA and full fine-tuning. Building upon this theoretical foundation, we propose LoRA-Pro: a method that designs a low-rank gradient calibration mechanism grounded in the theoretically optimal solution, employing a provably sound gradient reweighting strategy to enhance the representational capacity of low-rank updates. LoRA-Pro introduces no inference overhead—only learnable scaling factors are added. Extensive experiments across natural language understanding, dialogue generation, mathematical reasoning, code synthesis, and image classification demonstrate that LoRA-Pro significantly outperforms standard LoRA and closely approaches full fine-tuning performance. Our work provides both a novel theoretical perspective and a practical framework for parameter-efficient fine-tuning.

To address the substantial storage overhead and scalability challenges of LoRA modules in multi-task and personalized fine-tuning of large language models (LLMs), this paper proposes LoRA-XS—a radically lightweight low-rank adaptation method. Its core innovation lies in applying singular value decomposition (SVD) to pretrained weights, freezing the singular vectors, and introducing only a trainable $r imes r$ intermediate matrix—yielding an order-of-magnitude reduction in parameter count. This theory-driven architecture is the first to uncover structural redundancy in Transformer weight singular vectors while revealing their critical role in cross-task transfer. On a 7B model, LoRA-XS reduces parameters by over 100× compared to standard LoRA, yet matches or exceeds LoRA and VeRA performance on GLUE, GSM8K, MATH, and eight commonsense reasoning benchmarks. The method enables efficient deployment of million-scale personalized models.

This work addresses the performance degradation in gastrointestinal medical image recognition caused by distribution shifts across different data sources. To tackle this challenge, the authors propose a parameter-efficient fine-tuning approach based on Low-Rank Adaptation (LoRA). By introducing lightweight low-rank matrices into a pre-trained vision foundation model, the method adapts effectively to downstream gastrointestinal disease classification tasks while updating only a small fraction of parameters. Experimental results across multiple datasets demonstrate that LoRA fine-tuning achieves higher classification accuracy than full fine-tuning, despite using significantly fewer trainable parameters, thereby confirming its efficiency and superiority in domain adaptation for medical imaging.

This work addresses the trade-off between performance and efficiency in parameter-efficient fine-tuning of large language models by systematically reinterpreting Low-Rank Adaptation (LoRA) through the lens of signal processing. Leveraging classical low-rank modeling and inverse problem theory, it establishes a unified framework to understand both existing and future efficient fine-tuning methods. The study proposes a three-dimensional technical framework encompassing architecture design, optimization strategies, and full-lifecycle deployment, integrating core techniques such as singular value decomposition, rank expansion, cross-layer tensorization, norm-invariant optimization, and parameterization-aware solvers. This approach provides theoretical grounding and principled design guidelines for LoRA and its variants, while extending their applicability across pre-training, post-training, and deployment stages, thereby fostering bidirectional integration between signal processing and deep learning.

This work investigates whether low-rank adaptation (LoRA) can maintain strong generalization under structural constraints and proposes a more efficient, sparsity-aware fine-tuning approach. To this end, we introduce Cheap LoRA (cLA), along with its stochastic and cyclic chain variants, which enhance LoRA with sparsity. Theoretically, we establish the first information-theoretic generalization error bound for sparse LoRA and formalize cLA as a structured instance of asymmetric LoRA. Methodologically, our approach integrates sparse low-rank decomposition, randomly fixed-factor training, and cyclic parameterization. Extensive experiments across 10 pretrained models and 14 datasets demonstrate that cLA matches the performance of standard LoRA at equivalent parameter counts while reducing training time by up to 10% and peak GPU memory consumption by as much as 15%.

This work addresses the limitation of fixed-rank constraints in parameter-efficient fine-tuning, which fail to accommodate the heterogeneous rank requirements across different layers of neural networks. The authors propose LR-LoRA, a novel approach that introduces a learnable rank mechanism within the LoRA framework, enabling differentiable and dynamic optimization of the rank for each adapter layer. This method reveals a systematic disparity in rank demands between attention and MLP layers in Transformers, thereby providing a more flexible and effective inductive bias. Experimental results demonstrate that LR-LoRA significantly outperforms existing parameter-efficient fine-tuning methods across multiple benchmarks for language understanding and commonsense reasoning, achieving state-of-the-art performance.

This work proposes a sparse fine-tuning strategy based on layer importance assessment to address the issue of excessive trainable parameters in parameter-efficient fine-tuning of large language models. By leveraging similarity metrics such as Centered Kernel Alignment (CKA) to analyze representational changes across layers, the method identifies the most critical layers that contribute significantly to downstream tasks and applies LoRA or its variants exclusively to these layers. The approach is orthogonal and compatible with existing LoRA techniques, substantially reducing the number of trainable parameters. Experimental results across diverse benchmarks—including GLUE, mathematical reasoning, code generation, and multimodal tasks—demonstrate up to a 50% reduction in trainable parameters with negligible performance degradation or even slight improvements.