NP-LoRA: Null Space Projection Unifies Subject and Style in LoRA Fusion

2025-11-14

AI Summary

Existing LoRA fusion methods suffer from mutual interference between subject and style representations due to non-orthogonal and overlapping low-rank subspaces across modules, degrading generation fidelity and controllability. This work is the first to analyze this interference mechanism from the perspective of low-rank subspace structure. We propose a projection-based decoupled fusion framework: (i) orthogonal null-space projection—constructed via SVD—is applied to preserve the primary generative direction; (ii) a soft-projection strategy enables controllable injection of style components. Our method requires no retraining, is plug-and-play, and supports diverse backbone architectures and LoRA configurations. Extensive evaluations demonstrate significant improvements over mainstream baselines on DINO and CLIP metrics, as well as on human and large-model preference scores, validating its high fidelity, strong controllability, and broad compatibility.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-Rank Adaptation (LoRA) fusion has emerged as a key technique for reusing and composing learned subject and style representations for controllable generation without costly retraining. However, existing methods rely on weight-based merging, where one LoRA often dominates the other, leading to interference and degraded fidelity. This interference is structural: separately trained LoRAs occupy low-rank high-dimensional subspaces, leading to non-orthogonal and overlapping representations. In this work, we analyze the internal structure of LoRAs and find their generative behavior is dominated by a few principal directions in the low-rank subspace, which should remain free from interference during fusion. To achieve this, we propose Null Space Projection LoRA (NP-LoRA), a projection-based framework for LoRA fusion that enforces subspace separation to prevent structural interference among principal directions. Specifically, we first extract principal style directions via singular value decomposition (SVD) and then project the subject LoRA into its orthogonal null space. Furthermore, we introduce a soft projection mechanism that enables smooth control over the trade-off between subject fidelity and style consistency. Experiments show NP-LoRA consistently improves fusion quality over strong baselines (e.g., DINO and CLIP-based metrics, with human and LLM preference scores), and applies broadly across backbones and LoRA pairs without retraining.

Problem

• Prevents interference between subject and style LoRA representations during fusion
• Addresses structural overlap in low-rank subspaces of separately trained LoRAs
• Enables controlled trade-off between subject fidelity and style consistency

Innovation

• Projects LoRA into orthogonal null space
• Uses SVD to extract principal style directions
• Introduces soft projection for fidelity trade-off