Cross-LoRA: A Data-Free LoRA Transfer Framework across Heterogeneous LLMs

2025-08-07

AI Summary

Traditional parameter-efficient fine-tuning (PEFT) methods—e.g., LoRA—are tightly coupled with specific model architectures, hindering their transfer across heterogeneous large language models (LLMs). This work proposes LoRA-Align and LoRA-Shift, a two-component framework enabling **data-free, training-free** cross-architecture LoRA module transfer. LoRA-Align achieves source-to-target subspace alignment via truncated SVD and Frobenius-optimal linear transformation, while LoRA-Shift adapts the aligned LoRA weights through low-rank projection. The entire process completes in under 20 minutes on consumer-grade GPUs. Evaluated on commonsense reasoning benchmarks—including ARC, OBOA, and HellaSwag—the method yields up to +5.26% absolute improvement over base models, matching the performance of fully fine-tuned LoRA adapters. Crucially, it eliminates the need for task-specific data or retraining, substantially alleviating the generalization bottleneck of PEFT across diverse LLM architectures.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Traditional parameter-efficient fine-tuning (PEFT) methods such as LoRA are tightly coupled with the base model architecture, which constrains their applicability across heterogeneous pretrained large language models (LLMs). To address this limitation, we introduce Cross-LoRA, a data-free framework for transferring LoRA modules between diverse base models without requiring additional training data. Cross-LoRA consists of two key components: (a) LoRA-Align, which performs subspace alignment between source and target base models through rank-truncated singular value decomposition (SVD) and Frobenius-optimal linear transformation, ensuring compatibility under dimension mismatch; and (b) LoRA-Shift, which applies the aligned subspaces to project source LoRA weight updates into the target model parameter space. Both components are data-free, training-free, and enable lightweight adaptation on a commodity GPU in 20 minutes. Experiments on ARCs, OBOA and HellaSwag show that Cross-LoRA achieves relative gains of up to 5.26% over base models. Across other commonsense reasoning benchmarks, Cross-LoRA maintains performance comparable to that of directly trained LoRA adapters.

Problem

• Enables LoRA transfer across heterogeneous LLMs without data
• Aligns subspaces between models via SVD and linear transformation
• Projects source LoRA weights into target model space efficiently

Innovation

• Data-free LoRA transfer across diverse LLMs
• Subspace alignment via SVD and linear transformation
• Training-free lightweight adaptation on commodity GPUs