Label-Free Cross-Task LoRA Merging with Null-Space Compression

2026-03-27

AI Summary

This work proposes a label-free, output-type-agnostic method for merging Low-Rank Adaptation (LoRA) modules, addressing the limited generalization of existing approaches across diverse task types such as classification, regression, and generation. The key insight is the use of the degree of null-space compression in the LoRA down-projection matrix A during fine-tuning as a universal signal to guide a geometry-driven optimization of merging weights. This approach unifies support for vision classification, regression, sequence generation, and multimodal tasks. It achieves state-of-the-art performance across 20 heterogeneous vision tasks and significantly outperforms current methods on six natural language inference (NLI) benchmarks as well as vision-language tasks including VQA and image captioning.

Abstract

Model merging combines independently fine-tuned checkpoints without joint multi-task training. In the era of foundation-model, fine-tuning with Low-Rank Adaptation (LoRA) is prevalent, making LoRA merging a promising target. Existing approaches can work in homogeneous settings where all target tasks are classification but often fail when tasks span classification and regression. Approaches using entropy-based surrogates do not apply to regression and are costly for large language models due to long token sequences. We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry. Our key observation is that during LoRA finetuning the down-projection factor $A$ in $ΔW = BA$ compresses its null space, and the compression correlates with performance. NSC uses this as an optimization signal for merging that can generalize across classification, regression, and sequence generation. NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks. It also outperforms baselines on six NLI benchmarks and on vision-language evaluations for VQA and image captioning, demonstrating scalability and effectiveness.

Problem

• model merging
• LoRA
• heterogeneous tasks
• label-free
• cross-task

Innovation

• Null-Space Compression
• LoRA Merging
• Label-Free Fusion
• Cross-Task Generalization
• Adapter Geometry