RSLoRA: Training-free Rank Allocation for LoRA via Representational Sensitivity Probing

2026-07-05

AI Summary

This work addresses the limitation of conventional LoRA, which employs uniform rank allocation across layers despite their functional heterogeneity, and overcomes the drawbacks of existing adaptive rank methods—either computationally expensive or insufficient in capturing task-dependent dynamics of representation manifolds. The authors propose a novel, training- and gradient-free intelligent rank allocation strategy that simulates adaptation by injecting structured low-rank noise into activation spaces. By jointly leveraging effective rank and Fréchet distance to quantify manifold displacement, the method identifies modules most sensitive to representational shifts and allocates higher ranks accordingly. This approach introduces, for the first time, a representation sensitivity probing mechanism, achieving state-of-the-art performance over advanced baselines such as AdaLoRA and GoRA on mainstream benchmarks, while enabling efficient, robust, and representation-aware adaptation of large models.

Abstract

Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT); however, the conventional practice of uniform rank assignment ignores the functional heterogeneity of neural layers. Existing rank allocation methods typically struggle with a trade-off between computational intensity and heuristic simplicity: training-based methods suffer from prohibitive overhead, while pre-allocation methods fail to capture the dynamic task-specific representation manifold. In this paper, we propose RSLoRA (Representational Sensitivity LoRA), a training-free and gradient-free rank allocator driven by activation-space geometry. We identify a "sensitivity regime shift" across layers, observing that static weight analysis and local gradients are insufficient to reflect how updates reshape a model's internal representations. To address this, RSLoRA introduces a virtual representational probing mechanism. By simulating adaptation through structured low-rank noise and measuring the resulting manifold displacement by using Effective Rank and Frechet Distance, we identify high-sensitivity modules that require higher rank capacity. Our framework effectively bridges the gap between expert-crafted heuristics and actual representational impact. Extensive evaluations demonstrate that RSLoRA consistently outperforms state-of-the-art allocators (e.g., AdaLoRA, GoRA) across mainstream benchmarks. By eliminating the need for iterative training-time adjustments and backward gradients, RSLoRA provides a highly efficient, robust, and representation-aware solution for large-scale model adaptation.

Problem

• LoRA
• rank allocation
• parameter-efficient fine-tuning
• representational sensitivity
• training-free

Innovation

• Rank Allocation
• LoRA
• Representational Sensitivity
• Training-free
• Manifold Displacement