Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspace

2025-03-03

International Conference on Computational Linguistics

AI Summary

Low-rank adaptation methods (e.g., LoRA) are constrained by the rank-one assumption, limiting their ability to jointly optimize parameter efficiency and task adaptability. To address this, we propose Decoupled Convolutional Fine-Tuning (DCFT), the first PEFT method to incorporate learnable deconvolution: it dynamically reconstructs incremental parameters via tunable convolutional kernels within a low-dimensional subspace, thereby eliminating rigid low-rank constraints. DCFT decouples model parameter count from task complexity, enabling fine-grained, adaptive parameter modulation. Experiments demonstrate that DCFT reduces parameter count by 8× compared to LoRA while matching full fine-tuning performance across multiple downstream tasks. This substantially improves both the efficiency and generalization capability of large language model adaptation.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language model (LLM) is considered a milestone towards achieving Artificial General Intelligence (AGI). With its advanced emergent capabilities, it adapt to a wide range of specific applications. Fine-tuning LLMs for various downstream tasks has become a new paradigm. Low-Rank Adaptation (LoRA) is well-known for its parameter efficiency. It can reduce the number of parameters needed to fine-tune LLMs by several orders of magnitude. However, LoRA-based approaches encounter a significant limitation due to the bottleneck imposed by rank one decomposition. As the parameters count in LLMs increase, even rank one decomposition might surpass the number of parameters truly necessary for handling more downstream tasks. In this paper, we propose a new method for Parameter-Efficient Fine-Tuning (PEFT) via deconvolution in subspace, dubbed as DCFT. We innovatively use deconvolution to complete details and enhance knowledge in subspace incremental matrices, and dynamically control parameters by adjusting the kernel size, unconstrained by rank-one decomposition. Extensive experiments are conducted to validate the effectiveness of DCFT. Results show that compared to LoRA, DCFT achieve an 8$ imes$ reduction in parameters, and still achieves highly impressive performance. Our code is available here: https://github.com/Godz-z/DCFT.

Problem

• Efficient fine-tuning of large language models
• Overcoming limitations of rank-one decomposition
• Reducing parameters while maintaining high performance

Innovation

• Deconvolution in subspace for fine-tuning
• Dynamic parameter control via kernel size
• Achieves 8x parameter reduction over LoRA