Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

2026-05-07

AI Summary

This work addresses the trade-off between representational capacity and training efficiency in existing large language model fine-tuning methods, which typically choose between full fine-tuning (FFT) and low-rank adaptation (LoRA). The authors propose MoLF, a novel framework that enables dynamic, continuous mixing of FFT and LoRA at the optimizer level. MoLF employs a gradient-guided routing mechanism to precisely allocate parameter updates across both paths and integrates variable-rank LoRA pairs with a frozen backbone strategy, ensuring accurate gradient signals for both routes while maintaining memory efficiency. Experiments demonstrate that MoLF matches or nearly matches the best performance among FFT and LoRA baselines across all tasks (within ≤1.5% gap). Its efficient variant, MoLF-Efficient, surpasses current adaptive LoRA approaches by up to 20% on the Fact task and achieves gains of up to 9% on Med and SQL tasks.

Abstract

Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides the representational plasticity required for high-entropy knowledge injection, Low-Rank Adaptation (LoRA) can match or surpass FFT performance because many tasks only require updates in a low-rank space and benefit from LoRA's additional regularization. Through empirical evaluation across diverse tasks (SQL, Medical QA, and Counterfactual Knowledge) and varying language models (Gemma-3-1B, Qwen2.5-1.5B, and Qwen2.5-3B), we verify both trends and demonstrate that relying solely on either static architecture is structurally limited. To address this challenge, we propose a Mixture of LoRA and Full (MoLF) Fine-Tuning, a unified framework that enables continuous navigation between both training regimes. MoLF dynamically routes updates between FFT and LoRA at the optimizer level to ensure that exact gradient signals are available to both experts throughout training, yielding stable training dynamics. For memory-constrained environments, we also introduce MoLF-Efficient, which freezes base weights and only routes updates among a pair of LoRA experts of potentially varying rank. Our evaluations show that MoLF either improves on or stays within $1.5\%$ of the better of FFT and LoRA across all settings, while MoLF-Efficient outperforms prior adaptive LoRA approaches by up to $20\%$ on Fact and $9\%$ on Med and SQL.

Problem

• Large Language Models
• Fine-Tuning
• LoRA
• Full Fine-Tuning
• Parameter-Efficient Adaptation

Innovation

• Gradient-Guided Routing
• Mixture of Experts
• LoRA
• Full Fine-Tuning
• Parameter-Efficient Fine-Tuning