MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning

2025-03-27

AI Summary

Traditional LoRA employs a fixed rank for fine-tuning large language models, failing to adapt to hierarchical semantic structures—leading to parameter redundancy and suboptimal efficiency. To address this, we propose Multi-scale Pyramid LoRA (MP-LoRA), the first method to decouple parameter updates into three orthogonal scales: globally shared, mid-level shared, and layer-specific. MP-LoRA achieves cross-layer redundancy suppression and feature disentanglement via hierarchical low-rank decomposition and structured parameter sharing. Theoretical analysis grounded in Singular Value Decomposition (SVD) confirms its capacity for semantic separation. Empirically, on diverse NLP benchmarks, MP-LoRA reduces trainable parameters by 37% while improving average accuracy by 1.8% over standard LoRA—demonstrating superior information utilization efficiency and scalability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has become an essential approach for adapting large-scale pre-trained models while reducing computational costs. Among PEFT methods, LoRA significantly reduces trainable parameters by decomposing weight updates into low-rank matrices. However, traditional LoRA applies a fixed rank across all layers, failing to account for the varying complexity of hierarchical information, which leads to inefficient adaptation and redundancy. To address this, we propose MSPLoRA (Multi-Scale Pyramid LoRA), which introduces Global Shared LoRA, Mid-Level Shared LoRA, and Layer-Specific LoRA to capture global patterns, mid-level features, and fine-grained information, respectively. This hierarchical structure reduces inter-layer redundancy while maintaining strong adaptation capability. Experiments on various NLP tasks demonstrate that MSPLoRA achieves more efficient adaptation and better performance while significantly reducing the number of trainable parameters. Furthermore, additional analyses based on Singular Value Decomposition validate its information decoupling ability, highlighting MSPLoRA as a scalable and effective optimization strategy for parameter-efficient fine-tuning in large language models. Our code is available at https://github.com/Oblivioniss/MSPLoRA.

Problem

• Addresses inefficient adaptation in fixed-rank LoRA methods
• Reduces inter-layer redundancy while maintaining adaptation capability
• Improves parameter-efficient fine-tuning for large language models

Innovation

• Multi-scale pyramid structure for LoRA adaptation
• Hierarchical LoRA layers reduce parameter redundancy
• SVD-validated efficient information decoupling capability