Evolutionary Negative Module Pruning for Better LoRA Merging

2026-04-19

AI Summary

This work addresses a critical yet overlooked issue in existing LoRA merging methods: the implicit assumption that all adapter modules are beneficial, disregarding the presence of “negative modules” whose inclusion degrades performance after merging. The study is the first to identify and characterize this phenomenon and introduces ENMP, a plug-and-play evolutionary pruning method that identifies and removes harmful modules prior to merging. ENMP employs discrete, non-differentiable optimization to select modules via evolutionary search, making it compatible with mainstream LoRA merging algorithms and applicable across both language and vision models. Experimental results demonstrate that ENMP consistently yields significant performance gains across multiple tasks, achieving state-of-the-art merged model performance.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Merging multiple Low-Rank Adaptation (LoRA) experts into a single backbone is a promising approach for efficient multi-task deployment. While existing methods strive to alleviate interference via weight interpolation or subspace alignment, they rest upon the implicit assumption that all LoRA matrices contribute constructively to the merged model. In this paper, we uncover a critical bottleneck in current merging paradigms: the existence of $\textit{negative modules}$ -- specific LoRA layers that inherently degrade global performance upon merging. We propose $\textbf{E}$volutionary $\textbf{N}$egative $\textbf{M}$odule $\textbf{P}$runing ($\textbf{ENMP}$), a plug-and-play LoRA pruning method to locate and exclude these detrimental modules prior to merging. By leveraging an evolutionary search strategy, ENMP effectively navigates the discrete, non-differentiable landscape of module selection to identify optimal pruning configurations. Extensive evaluations demonstrate that ENMP consistently boosts the performance of existing merging algorithms, achieving a new state-of-the-art across both language and vision domains. Code is available at https://github.com/CaoAnda/ENMP-LoRAMerging.

Problem

• LoRA merging
• negative modules
• multi-task deployment
• model interference
• module pruning

Innovation

• LoRA merging
• negative module pruning
• evolutionary search
• multi-task adaptation
• parameter-efficient fine-tuning