Accurate and Efficient Low-Rank Model Merging in Core Space

2025-09-22

AI Summary

To address the challenge of balancing efficiency and cross-task accuracy in LoRA model merging, this paper proposes Core Space: a novel framework that constructs a shared alignment basis space, enabling lossless projection of task-specific LoRA weights into a unified low-dimensional core subspace prior to fusion. We theoretically prove that this projection preserves the original rank and information integrity—thereby avoiding representational collapse inherent in conventional weighted averaging. Core Space integrates low-rank adaptation, orthogonal projection-based alignment, and computational complexity optimization to achieve efficient weight aggregation within the core subspace. Evaluated on diverse vision and language multi-task benchmarks, Core Space achieves state-of-the-art performance while incurring only ~20% of the computational overhead of existing merging methods. It thus delivers superior accuracy, resource efficiency, and strong generalization across tasks.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.

Problem

• Merging low-rank adapted neural networks efficiently without sacrificing accuracy
• Preserving adaptation efficiency while improving merging accuracy across multiple tasks
• Enabling model merging in common alignment basis to reduce computational resources

Innovation

• Merging models in Core Space alignment basis
• Preserving efficiency while improving accuracy
• Ensuring no information loss via projection