CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models

2025-10-11

AI Summary

To address the misalignment between parameter capacity allocation and gradient update dynamics in Parameter-Efficient Fine-Tuning (PEFT)—which causes training instability and limited generalization—this paper proposes CurvLoRA, a curvature-aware, trust-region-guided low-rank adaptation framework. CurvLoRA jointly optimizes dynamic rank allocation and update stability: it employs lightweight second-order information (Fisher/Hessian approximations) to drive adaptive rank scheduling and enforces trust-region constraints to ensure reliable gradient updates. Evaluated on open-source large language models (7B–13B parameters), CurvLoRA consistently outperforms mainstream PEFT methods—including LoRA and AdaLoRA—achieving higher downstream task accuracy and improved convergence stability, while simultaneously reducing memory footprint and increasing training throughput. The method thus attains a Pareto-optimal trade-off between performance and efficiency.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Parameter-efficient fine-tuning (PEFT) has become the standard approach for adapting large language models under limited compute and memory budgets. Although previous methods improve efficiency through low-rank updates, quantization, or heuristic budget reallocation, they often decouple the allocation of capacity from the way updates evolve during training. In this work, we introduce CTR-LoRA, a framework guided by curvature trust region that integrates rank scheduling with stability-aware optimization. CTR-LoRA allocates parameters based on marginal utility derived from lightweight second-order proxies and constrains updates using a Fisher/Hessian-metric trust region. Experiments on multiple open-source backbones (7B-13B), evaluated on both in-distribution and out-of-distribution benchmarks, show consistent improvements over strong PEFT baselines. In addition to increased accuracy, CTR-LoRA enhances training stability, reduces memory requirements, and achieves higher throughput, positioning it on the Pareto frontier of performance and efficiency. These results highlight a principled path toward more robust and deployable PEFT.

Problem

• Optimizes parameter-efficient fine-tuning for large language models
• Integrates rank scheduling with stability-aware optimization methods
• Improves accuracy while reducing memory requirements and enhancing stability

Innovation

• Curvature trust region guides low-rank adaptation
• Rank scheduling integrates with stability-aware optimization
• Parameter allocation uses second-order marginal utility