AILoRA: Function-Aware Asymmetric Initialization for Low-Rank Adaptation of Large Language Models

2025-10-09

AI Summary

To address suboptimal performance and slow convergence of Low-Rank Adaptation (LoRA) in fine-tuning large language models, this paper proposes a function-aware asymmetric LoRA initialization. We first identify the functional asymmetry between query (W^Q) and value (W^V) projection matrices in self-attention: W^Q primarily governs task-specific modeling, whereas W^V emphasizes general feature extraction. Leveraging this insight, we introduce a principal–auxiliary component co-initialization strategy—assigning high signal-to-noise-ratio principal components to W^Q-LoRA to enhance task adaptation, while retaining low-rank auxiliary components for W^V-LoRA to preserve representational generality. Our method operates within standard LoRA architecture and enables efficient deployment via singular value decomposition (SVD). Experiments demonstrate that, with minimal parameter overhead (<0.1%), our approach accelerates convergence by 1.8× on average and improves downstream task accuracy by +2.3% on average, consistently outperforming vanilla LoRA.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Parameter-efficient finetuning (PEFT) aims to mitigate the substantial computational and memory overhead involved in adapting large-scale pretrained models to diverse downstream tasks. Among numerous PEFT strategies, Low-Rank Adaptation (LoRA) has emerged as one of the most widely adopted approaches due to its robust empirical performance and low implementation complexity. In practical deployment, LoRA is typically applied to the $W^Q$ and $W^V$ projection matrices of self-attention modules, enabling an effective trade-off between model performance and parameter efficiency. While LoRA has achieved considerable empirical success, it still encounters challenges such as suboptimal performance and slow convergence. To address these limitations, we introduce extbf{AILoRA}, a novel parameter-efficient method that incorporates function-aware asymmetric low-rank priors. Our empirical analysis reveals that the projection matrices $W^Q$ and $W^V$ in the self-attention mechanism exhibit distinct parameter characteristics, stemming from their functional differences. Specifically, $W^Q$ captures task-specific semantic space knowledge essential for attention distributions computation, making its parameters highly sensitive to downstream task variations. In contrast, $W^V$ encodes token-level feature representations that tend to remain stable across tasks and layers. Leveraging these insights, AILoRA performs a function-aware initialization by injecting the principal components of $W^Q$ to retain task-adaptive capacity, and the minor components of $W^V$ to preserve generalizable feature representations. This asymmetric initialization strategy enables LoRA modules to better capture the specialized roles of attention parameters, thereby enhancing both finetuning performance and convergence efficiency.

Problem

• Enhances LoRA performance by addressing suboptimal results and slow convergence
• Introduces function-aware asymmetric initialization for attention projection matrices
• Leverages distinct parameter characteristics of W^Q and W^V matrices

Innovation

• Function-aware asymmetric low-rank priors initialization
• Principal components injection for task-adaptive capacity
• Minor components injection for generalizable representations