Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

2025-07-03

AI Summary

Addressing the trade-off between efficiency and representational capacity in continual fine-tuning of large language models (LLMs), this paper proposes GORP: a method that jointly optimizes full-model parameters and low-rank parameters within a unified low-rank gradient subspace, eliminating LoRA’s reliance on explicit parameter constraints. By projecting gradients onto a low-rank subspace and enforcing subspace-aware optimization, GORP expands the effective optimization space without increasing training overhead—thereby enhancing knowledge transfer and mitigating catastrophic forgetting. Experiments across multiple continual learning benchmarks demonstrate that GORP consistently outperforms existing state-of-the-art methods, achieving superior generalization and task adaptability while preserving efficient fine-tuning. GORP establishes a new paradigm for LLM continual learning that is both theoretically grounded and practically deployable.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness. Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. We propose GORP (Gradient LOw Rank Projection) for Continual Learning, a novel training strategy that overcomes these limitations by synergistically combining full and low-rank parameters and jointly updating within a unified low-rank gradient subspace. GORP expands the optimization space while preserving efficiency and mitigating catastrophic forgetting. Extensive experiments on continual learning benchmarks demonstrate GORP's superior performance compared to existing state-of-the-art approaches. Code is available at https://github.com/Wcxwcxw/GORP.

Problem

• Balances efficiency and expressiveness in continual LLM fine-tuning
• Overcomes LoRA's limitations in learning new tasks and knowledge transfer
• Expands optimization space while preventing catastrophic forgetting

Innovation

• Combines full and low-rank parameters synergistically
• Updates within unified low-rank gradient subspace
• Expands optimization space preserving efficiency