Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging

2025-05-28

AI Summary

Low-rank adaptation (LoRA) fine-tuned models suffer significant performance degradation during multi-task merging due to parameter interference. Method: This paper first identifies a novel interference mechanism arising from the interaction between data distributions and LoRA parameters, and proposes OSRM—a pre-merging orthogonal subspace regularization framework. OSRM enforces orthogonality among the low-rank subspaces of LoRA adapters *before* fine-tuning, thereby decoupling tasks and enhancing merging robustness. It integrates orthogonal constraints, standard LoRA fine-tuning, mainstream merging algorithms (TIES-Merging and DARE), and projection-based optimization. Results: Extensive experiments across eight datasets and five large language models demonstrate that OSRM substantially improves multi-task merging accuracy while preserving single-task performance with zero degradation. Moreover, OSRM exhibits superior robustness to merging hyperparameters compared to baselines.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Fine-tuning large language models (LMs) for individual tasks yields strong performance but is expensive for deployment and storage. Recent works explore model merging to combine multiple task-specific models into a single multi-task model without additional training. However, existing merging methods often fail for models fine-tuned with low-rank adaptation (LoRA), due to significant performance degradation. In this paper, we show that this issue arises from a previously overlooked interplay between model parameters and data distributions. We propose Orthogonal Subspaces for Robust model Merging (OSRM) to constrain the LoRA subspace *prior* to fine-tuning, ensuring that updates relevant to one task do not adversely shift outputs for others. Our approach can seamlessly integrate with most existing merging algorithms, reducing the unintended interference among tasks. Extensive experiments on eight datasets, tested with three widely used LMs and two large LMs, demonstrate that our method not only boosts merging performance but also preserves single-task accuracy. Furthermore, our approach exhibits greater robustness to the hyperparameters of merging. These results highlight the importance of data-parameter interaction in model merging and offer a plug-and-play solution for merging LoRA models.

Problem

• Address performance degradation in LoRA model merging
• Mitigate interference between tasks in merged models
• Ensure robust merging without additional training

Innovation

• Orthogonal Subspaces for Robust Merging
• Prevents LoRA interference via subspace constraints
• Plug-and-play solution for LoRA model merging