fine-tuning

To address the insufficient generalization of pretrained models when fine-tuned on few-shot, high-dimensional binary classification tasks, this paper proposes a novel fine-tuning framework based on weight matrix reparameterization. The method couples low-rank adaptation (LoRA) with a base-model rescaling mechanism and employs random matrix theory to model the generalization behavior of high-dimensional classifiers, thereby revealing how rescaling governs spectral distribution and generalization bounds. Theoretically, the approach substantially mitigates overfitting by controlling the effective rank and condition number of the classifier’s weight matrix. Empirically, it consistently improves performance across multiple binary classification benchmarks and large language model (LLM) fine-tuning tasks, demonstrating particularly pronounced generalization gains under extreme data scarcity.

This work addresses catastrophic forgetting in fine-tuning pretrained models, where newly acquired knowledge overwrites previously learned information. To mitigate this issue, the authors propose a function-preserving model expansion approach that mathematically duplicates and scales parameters of selected Transformer submodules during initialization. This technique enables stable training and faithful retention of original model capabilities without altering the initial functionality. By circumventing the traditional trade-off between plasticity and stability, the method achieves performance comparable to full fine-tuning while expanding only a minimal number of layers. Consequently, it fully preserves the model’s original knowledge and substantially reduces computational overhead.

Parameter-efficient fine-tuning (PEFT) methods such as LoRA often induce catastrophic forgetting of pre-trained general capabilities in large language models during instruction tuning—especially under few-shot settings. To address this, we propose Approximate Regularized Replay (ARR), the first approach that jointly leverages KL-divergence regularization and lightweight, cross-corpus next-token replay. ARR interleaves samples from multiple data sources to simultaneously optimize knowledge retention and task adaptation with minimal overhead. Evaluated on the Qwen family of models, ARR preserves new-task performance while substantially mitigating capability degradation; pre-trained knowledge integrity improves markedly, with only ~15% additional computational cost. This work provides an efficient, general-purpose, and deployment-friendly solution to the continual learning challenge in PEFT.

To address the prominent GPU memory bottleneck in large language model (LLM) pretraining, this paper proposes the Staged Parameter-Efficient Training (SPET) framework. SPET is the first to deeply integrate parameter-efficient fine-tuning techniques—such as LoRA—into the *entire* pretraining pipeline, synergistically combining gradient checkpointing with staged architectural expansion to enable dynamic model growth and on-demand memory optimization. Implemented in PyTorch, SPET introduces a memory-aware training scheduler that reduces peak GPU memory consumption by up to 53.9% versus full-parameter baselines, while preserving pretraining performance. Downstream task performance after instruction tuning remains unchanged. The core contribution lies in bridging the paradigmatic divide between standard pretraining and parameter-efficient adaptation, establishing a scalable, memory-efficient, and unified pretraining paradigm.

This work investigates how low-rank adaptation (LoRA) parameters influence catastrophic forgetting during fine-tuning. We systematically merge LoRA adapter weights back into the backbone to quantitatively analyze forgetting dynamics across pretraining and downstream tasks, as well as changes in model plasticity. We identify, for the first time, a “contextual forgetting” phenomenon in Vision Transformers (ViTs)—characterized by task-dependent degradation of local features—distinct from the global forgetting observed in ResNets and unreported in prior continual learning literature. Moreover, we reveal that LoRA rank exerts a dual regulatory effect: excessively low ranks exacerbate pretrained knowledge forgetting, while excessively high ranks impair downstream adaptability. Experiments span diverse multi-task continual learning scenarios. Our findings provide theoretical foundations and principled guidelines for rank selection in efficient, sustainable visual model adaptation.

This work addresses the lack of theoretical understanding regarding the generalization gap between Low-Rank Adaptation (LoRA) and full fine-tuning. Within a simplified linear regression framework, the study systematically compares their generalization behaviors by integrating statistical learning theory with excess risk analysis. It establishes, for the first time, a theoretical guarantee that when the discrepancy between the pre-trained model and the downstream task exhibits a low-rank structure, LoRA achieves lower excess risk than full fine-tuning in both over-parameterized and under-parameterized regimes. Empirical experiments further corroborate this finding, demonstrating a non-intuitive improvement in test accuracy under low-rank constraints and thereby revealing the intrinsic mechanism underlying LoRA’s superior generalization capability.

Existing theoretical frameworks struggle to explain why larger-scale pre-trained models substantially reduce sample complexity on downstream tasks. This work proposes a novel theoretical framework—termed “caulking”—inspired by parameter-efficient fine-tuning methods such as adapters, low-rank adaptation, and partial fine-tuning. It establishes, for the first time, a provable relationship between the scale of pre-trained models and the sample complexity of downstream tasks. By rigorously linking stronger pre-training capabilities to reduced data requirements in transfer learning, this study not only addresses a critical gap in current theoretical understanding but also provides a solid foundation for empirically observed scaling laws, demonstrating that enhanced pre-training capacity can significantly decrease the number of samples needed for effective downstream adaptation.

This work addresses the high computational cost of conventional fine-tuning for instance segmentation, which typically requires updating a large fraction (40–55%) of parameters in large pre-trained models. To improve parameter efficiency, the study explores parameter-efficient fine-tuning (PEFT) methods, introducing LoRA into deformable attention mechanisms for the first time and systematically evaluating the trade-offs between performance and efficiency based on the number and placement of adapters within the Transformer architecture. Experimental results demonstrate that by fine-tuning only 1–6% of the model parameters, the proposed approach matches or even surpasses the performance of full fine-tuning across four benchmark datasets. These findings validate the efficacy and feasibility of PEFT for instance segmentation and further reveal that its effectiveness is influenced by dataset complexity and model architecture.

This study investigates whether pretraining always benefits LoRA fine-tuning, revealing that excessive pretraining can paradoxically slow convergence. By constructing a single-index model and analyzing LoRA fine-tuning dynamics under single-pass stochastic gradient descent (SGD), the work characterizes how convergence speed is jointly governed by the initial alignment between pretrained and target tasks and the nonlinearity of the target task. Notably, it demonstrates—through the lens of optimization dynamics—that even with strong task alignment, aggressive pretraining may induce a prolonged search phase, impeding efficient convergence of LoRA. The paper establishes a unified theoretical framework that precisely describes fine-tuning behavior as a function of both pretraining strength and task complexity, challenging the prevailing intuition that “stronger pretraining is always better” and offering new insights for designing efficient parameter-efficient fine-tuning strategies.

This work addresses the limitation of fixed-rank constraints in parameter-efficient fine-tuning, which fail to accommodate the heterogeneous rank requirements across different layers of neural networks. The authors propose LR-LoRA, a novel approach that introduces a learnable rank mechanism within the LoRA framework, enabling differentiable and dynamic optimization of the rank for each adapter layer. This method reveals a systematic disparity in rank demands between attention and MLP layers in Transformers, thereby providing a more flexible and effective inductive bias. Experimental results demonstrate that LR-LoRA significantly outperforms existing parameter-efficient fine-tuning methods across multiple benchmarks for language understanding and commonsense reasoning, achieving state-of-the-art performance.