LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization

2025-07-06

AI Summary

To address the high computational overhead, limited expressivity, and catastrophic forgetting inherent in low-rank adaptation (LoRA) and similar methods for domain specialization and continual learning, this paper proposes LoSiA: a dynamic subnet localization and high-rank adaptation framework grounded in gradient sparsity analysis. LoSiA abandons fixed low-rank parameterization and instead identifies sparse, task-critical trainable subnetworks via gradient sparsity patterns, applying high-rank updates exclusively to these subnets. We further introduce LoSiA-Pro, an optimized implementation that reduces training latency. Experiments demonstrate that LoSiA achieves performance on par with full fine-tuning on domain adaptation and commonsense reasoning benchmarks, while attaining the fastest training speed—reducing training latency by approximately 27% compared to LoRA—and significantly mitigating knowledge forgetting during continual learning.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, significantly reduce the number of trainable parameters by introducing low-rank decomposition matrices. However, existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. Hence, we propose LoSiA(Low-Resources Subnet Integration Adaptation), an innovative method that dynamically localizes and optimizes critical parameters during the training process. Specifically, it identifies a sub-network using gradient sparsity analysis and optimizes it as the trainable target. This design enables effective high-rank adaptation by updating only the sub-network parameters, reducing the additional matrix multiplication. We also present LoSiA-Pro, a faster implementation of LoSiA, which reduces the training latency by about $27%$ compared to LoRA. Extensive evaluations show that our method achieves minimal performance drop compared to full fine-tuning, while requiring the least training time across domain specialization and common-sense reasoning tasks. Further analysis shows that LoSiA also reduces forgetting during continued training.

Problem

• Reduces computational inefficiency in high-rank fine-tuning
• Improves fine-tuning performance via dynamic subnet optimization
• Minimizes training time while maintaining model accuracy

Innovation

• Dynamic subnet localization via gradient sparsity
• Optimizes only critical sub-network parameters
• Reduces matrix multiplications for efficiency