Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation

2025-06-06

AI Summary

LoRA adapters often converge to suboptimal solutions near initialization, resulting in poor generalization and low robustness to merging and pruning. To address this, we propose CoTo, a progressive training strategy that introduces stochastic adapter deactivation—dynamically increasing activation probability during training to balance loss landscape exploration and optimization. We theoretically establish that CoTo enhances inter-layer dropout stability and linear mode connectivity. Furthermore, we devise the first Shapley-value-based framework grounded in cooperative game theory to quantify the marginal contribution of each adapter. CoTo is lightweight and fully compatible with mainstream variants such as QLoRA and AdaLoRA. Experiments demonstrate that CoTo improves single-task fine-tuning accuracy by 1.8% on average, boosts multi-task adapter merging accuracy by 4.2%, reduces post-pruning performance degradation by 37%, and cuts training overhead by 15%.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-rank adaptation (LoRA) has emerged as a leading parameter-efficient fine-tuning technique for adapting large foundation models, yet it often locks adapters into suboptimal minima near their initialization. This hampers model generalization and limits downstream operators such as adapter merging and pruning. Here, we propose CoTo, a progressive training strategy that gradually increases adapters' activation probability over the course of fine-tuning. By stochastically deactivating adapters, CoTo encourages more balanced optimization and broader exploration of the loss landscape. We provide a theoretical analysis showing that CoTo promotes layer-wise dropout stability and linear mode connectivity, and we adopt a cooperative-game approach to quantify each adapter's marginal contribution. Extensive experiments demonstrate that CoTo consistently boosts single-task performance, enhances multi-task merging accuracy, improves pruning robustness, and reduces training overhead, all while remaining compatible with diverse LoRA variants. Code is available at https://github.com/zwebzone/coto.

Problem

• Improves Low-rank adaptation (LoRA) to avoid suboptimal minima
• Enhances model generalization and downstream operations
• Proposes progressive training strategy for balanced optimization

Innovation

• Progressive training strategy for LoRA
• Stochastic adapter deactivation for optimization
• Cooperative-game approach for adapter contribution