EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition

2025-01-21

AI Summary

Existing parameter-efficient fine-tuning (PEFT) methods—such as LoRA—suffer from poor scalability and divergent learning dynamics relative to full fine-tuning. To address this, we propose a lightweight fine-tuning framework based on weight decoupling: pretrained weights are decomposed into magnitude and direction components; a low-rank basis matrix initialized via singular value decomposition (SVD) is frozen, while only a compact adapter matrix is optimized. This design preserves strong representational capacity despite extremely low parameter counts. On the GLUE benchmark, our method matches or surpasses LoRA and DoRA in performance while reducing trainable parameters by up to 30×, significantly enhancing adaptation efficiency for large language models under memory constraints. The core innovation lies in the first integration of weight decoupling modeling with SVD-based basis freezing—achieving a principled unification of high accuracy and high efficiency in PEFT.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Parameter-efficient fine-tuning methods, such as LoRA, reduces the number of trainable parameters. However, they often suffer from scalability issues and differences between their learning pattern and full fine-tuning. To overcome these limitations, we propose Efficient Weight-Decomposed Low-Rank Adaptation (EDoRA): a novel PEFT method that decomposes pre-trained weights into magnitude and directional components. By freezing low-rank matrices, initializing them by singular value decomposition, and introducing a small trainable matrix between them, EDoRA achieves substantial reduction in trainable parameters while maintaining learning capacity. Experimental results on the GLUE benchmark demonstrate that EDoRA achieves competitive or superior performance compared to state-of-the-art methods, such as LoRA and DoRA, with up to 30x fewer trainable parameters. This makes EDoRA a highly efficient solution for adapting LLMs to diverse tasks under memory-constrained settings. Code is available at https://github.com/Hamid-Nasiri/EDoRA .

Problem

• Large-scale Problems
• LoRA Efficiency
• Resource-limited Environment

Innovation

• EDoRA
• Parameter Reduction
• SVD-based Decomposition