Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

2026-05-12

AI Summary

This study investigates how to select trainable parameters under a fixed parameter budget in Low-Rank Adaptation (LoRA) to maximize model performance. It reveals that the effectiveness of parameter selection strategies is highly dependent on the training paradigm: under supervised fine-tuning, random and gradient-guided selection perform comparably, whereas only gradient-guided selection yields gains in GRPO-based reinforcement learning fine-tuning. Building on this insight, the authors propose an efficient scoring method that leverages gradient structure to rapidly identify critical parameters. This approach requires less than 0.5% of the full training cost and completes selection within ten seconds, consistently improving performance across models ranging from 1.5B to 8B parameters. The identified critical parameters are predominantly concentrated in the value (V), output (O), and down-projection matrices.

Abstract

We study the \textit{parameter placement problem}: given a fixed budget of $k$ trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which $k$ matter? Under supervised fine-tuning, random and informed subsets achieve comparable performance. Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy. This regime dependence traces to gradient structure: SFT gradients are low-rank and directionally stable, so any subset accumulates coherent updates; GRPO gradients are high-rank and near-orthogonal across steps, so only elements with consistently signed gradients retain the learning signal. Our scoring procedure identifies these critical parameters in under 10 seconds at less than 0.5% of training cost. Selected parameters concentrate on residual-stream-writing projections (V, O, Down), stable across model families and scales (1.5B - 8B).

Problem

• parameter placement
• Low-Rank Adaptation
• LoRA
• gradient structure
• fine-tuning

Innovation

• parameter placement
• LoRA
• gradient structure
• low-rank adaptation
• efficient fine-tuning