Mixture-of-Subspaces in Low-Rank Adaptation

2024-06-16

Conference on Empirical Methods in Natural Language Processing

AI Summary

To address the limited generalization of Low-Rank Adaptation (LoRA) in multimodal tasks, this paper proposes MoSLoRA—a novel subspace-aware LoRA variant. MoSLoRA reformulates LoRA by decomposing the weight matrix into two orthogonal subspaces and introducing a learnable linear mixer for dynamic fusion. Crucially, it jointly optimizes both subspace representations and mixing coefficients without increasing inference overhead and remains fully modality-agnostic. Extensive experiments across three diverse multimodal tasks—commonsense reasoning, vision-instruction fine-tuning, and topic-driven text-to-image generation—demonstrate consistent and significant improvements over standard LoRA, with substantial average gains. These results validate MoSLoRA’s effectiveness and cross-modal robustness. The core contribution lies in the synergistic design of subspace decoupling and learnable mixing, establishing a new paradigm for parameter-efficient fine-tuning.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA) method, which is computationally efficient, easy to implement, and readily applicable to large language, multimodal, and diffusion models. Initially, we equivalently decompose the weights of LoRA into two subspaces, and find that simply mixing them can enhance performance. To study such a phenomenon, we revisit it through a fine-grained subspace lens, showing that such modification is equivalent to employing a fixed mixer to fuse the subspaces. To be more flexible, we jointly learn the mixer with the original LoRA weights, and term the method as Mixture-of-Subspaces LoRA (MoSLoRA). MoSLoRA consistently outperforms LoRA on tasks in different modalities, including commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation, demonstrating its effectiveness and robustness.

Problem

• Enhance performance of Low-Rank Adaptation (LoRA) via subspace mixing.
• Introduce Mixture-of-Subspaces LoRA (MoSLoRA) for flexible subspace fusion.
• Demonstrate MoSLoRA's effectiveness across multimodal tasks and robustness.

Innovation

• Subspace-inspired Low-Rank Adaptation method
• Joint learning of mixer with LoRA weights
• Mixture-of-Subspaces LoRA enhances multimodal tasks