Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectiveness

2025-06-16

AI Summary

This work challenges the generalization capability of Low-Rank Adaptation (LoRA) reuse in model merging and routing, arguing that it often achieves only shallow pattern matching rather than genuine compositional generalization. Method: Through theoretical analysis and controlled experiments—including synthetic two-hop reasoning and mathematical word problems—we systematically investigate LoRA’s reuse behavior under varying pretraining knowledge coverage. Contribution/Results: We identify a fundamental limitation: LoRA reuse efficacy degrades sharply when pretrained representations are insufficient for cross-dataset logical integration. We formally characterize the feasibility boundary of LoRA as a zero-data reuse method, demonstrating that its success critically depends on the completeness of underlying pretrained knowledge. Crucially, we advocate prioritizing rigorous evaluation frameworks over novel algorithm design. Our core contribution is establishing verifiable theoretical constraints on LoRA reuse—providing both a benchmark and principled guidance for future research.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Merging or routing low-rank adapters (LoRAs) has emerged as a popular solution for enhancing large language models, particularly when data access is restricted by regulatory or domain-specific constraints. This position paper argues that the research community should shift its focus from developing new merging or routing algorithms to understanding the conditions under which reusing LoRAs is truly effective. Through theoretical analysis and synthetic two-hop reasoning and math word-problem tasks, we examine whether reusing LoRAs enables genuine compositional generalization or merely reflects shallow pattern matching. Evaluating two data-agnostic methods--parameter averaging and dynamic adapter selection--we found that reusing LoRAs often fails to logically integrate knowledge across disjoint fine-tuning datasets, especially when such knowledge is underrepresented during pretraining. Our empirical results, supported by theoretical insights into LoRA's limited expressiveness, highlight the preconditions and constraints of reusing them for unseen tasks and cast doubt on its feasibility as a truly data-free approach. We advocate for pausing the pursuit of novel methods for recycling LoRAs and emphasize the need for rigorous mechanisms to guide future academic research in adapter-based model merging and practical system designs for practitioners.

Problem

• Examine LoRA reuse effectiveness for compositional generalization
• Evaluate data-agnostic methods for integrating disjoint knowledge
• Identify preconditions and constraints of LoRA reuse

Innovation

• Shift focus from LoRA merging to effectiveness conditions
• Evaluate LoRA reuse via theoretical and synthetic tasks
• Highlight constraints of LoRA reuse for unseen tasks