Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models

2025-09-10

AI Summary

To address the inefficiency and instability of fixed-rank LoRA in resource-constrained large language model (LLM) adaptation, this paper proposes SensLoRA, a sensitivity-aware dynamic low-rank fine-tuning method. SensLoRA jointly leverages global and local parameter sensitivity analysis, efficiently estimating update importance via diagonal approximations of the loss Hessian, and adaptively allocates rank per layer or module accordingly. Unlike existing rank-allocation strategies, SensLoRA is computationally lightweight, training-stable, and deployment-friendly. Extensive experiments across multiple downstream tasks and benchmarks demonstrate that SensLoRA significantly reduces trainable parameters, GPU memory consumption, and training time compared to standard LoRA and state-of-the-art variants—while maintaining or even improving task performance. Thus, SensLoRA achieves a unified optimization of efficiency, stability, and effectiveness in LLM fine-tuning under resource constraints.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have transformed both everyday life and scientific research. However, adapting LLMs from general-purpose models to specialized tasks remains challenging, particularly in resource-constrained environments. Low-Rank Adaptation (LoRA), a prominent method within Parameter-Efficient Fine-Tuning (PEFT), has emerged as a promising approach to LLMs by approximating model weight updates using low-rank decomposition. However, LoRA is limited by its uniform rank ( r ) allocation to each incremental matrix, and existing rank allocation techniques aimed at addressing this issue remain computationally inefficient, complex, and unstable, hindering practical applications. To address these limitations, we propose Sensitivity-LoRA, an efficient fine-tuning method that dynamically allocates ranks to weight matrices based on both their global and local sensitivities. It leverages the second-order derivatives (Hessian Matrix) of the loss function to effectively capture weight sensitivity, enabling optimal rank allocation with minimal computational overhead. Our experimental results have demonstrated robust effectiveness, efficiency and stability of Sensitivity-LoRA across diverse tasks and benchmarks.

Problem

• Dynamically allocates ranks based on sensitivity
• Overcomes uniform rank limitation in LoRA
• Reduces computational complexity of fine-tuning

Innovation

• Dynamic rank allocation based on sensitivity
• Uses Hessian Matrix for weight sensitivity analysis
• Minimal computational overhead for optimal performance