Separating Shared and Domain-Specific LoRAs for Multi-Domain Learning

2025-08-04

AI Summary

Existing multi-domain LoRA architectures suffer from representational coupling between shared and domain-specific adapters, undermining domain specificity. To address this, we propose a subspace orthogonalization constraint mechanism that strictly confines the shared LoRA to the column space of the pre-trained weight matrix, while constraining domain-specific LoRAs to its left null space—achieving geometric decoupling. Our method integrates low-rank adaptation with matrix subspace decomposition and is evaluated on joint training across UCF101, Kinetics-400, and HMDB51 for action recognition. Experiments demonstrate significant improvements in multi-domain generalization and domain discriminability. Furthermore, dimensionality analysis of LoRA weights reveals a more interpretable and well-defined representational division of labor between shared and domain-specific components. This work constitutes the first approach to enforce strict subspace orthogonality between shared and domain-specific LoRA modules, thereby enhancing both parameter efficiency and domain-aware representation learning.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Representation Learning for Vision
• Application Domains: Mobility, Driving & Flight

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Existing architectures of multi-domain learning have two types of adapters: shared LoRA for all domains and domain-specific LoRA for each particular domain. However, it remains unclear whether this structure effectively captures domain-specific information. In this paper, we propose a method that ensures that shared and domain-specific LoRAs exist in different subspaces; specifically, the column and left null subspaces of the pre-trained weights. We apply the proposed method to action recognition with three datasets (UCF101, Kinetics400, and HMDB51) and demonstrate its effectiveness in some cases along with the analysis of the dimensions of LoRA weights.

Problem

• Separating shared and domain-specific LoRAs effectively
• Ensuring LoRAs exist in different subspaces
• Validating method on action recognition datasets

Innovation

• Separates shared and domain-specific LoRAs
• Ensures LoRAs exist in different subspaces
• Applies method to action recognition datasets