Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update

2025-09-24

AI Summary

While LoRA fine-tuning reduces parameter count and memory usage, it lags behind full low-rank training (SVDLoRA) in performance. Method: We propose OPLoRA, a memory-efficient optimizer based on Alternating Least Squares (ALS) that decouples LoRA optimization into interpretable subproblems; it approximates truncated SVD accuracy within 1–2 iterations without explicitly forming large matrices. OPLoRA is the first to formulate LoRA training as an alternating update framework, unifying preconditioning strategies and introducing a low-rank-estimation-driven momentum mechanism. Contribution/Results: With only ~3× the parameter overhead of standard LoRA, OPLoRA significantly narrows the performance gap with SVDLoRA. It achieves lower memory consumption and superior generalization across diverse benchmarks—including MNIST, CIFAR-100, and RoBERTa-base—demonstrating both efficiency and effectiveness.

Technology Category

• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. However, there is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved. In this study, we propose OPLoRA, a memory-efficient optimizer that closes this gap by casting LoRA optimization as an interpretable sub-problem and solving it efficiently with alternating least squares updates, where 1-2 alternating steps are empirically found to be sufficient to closely match truncated SVD without ever forming the full matrix. We also retrieve the recently proposed preconditioning methods for LoRA as a special case. OPLoRA supports momentum by maintaining a low-rank estimate using the same subroutine (LoRSum) for computing the step, with a memory budget of 3 times the number of LoRA parameters (i.e., same as Adam). We also propose an experimental scaled variant that uses the K-FAC metric, which could be of interest. Across a linear task, MNIST, CIFAR-100, and RoBERTa-base (MNLI), OPLoRA consistently approaches SVDLoRA's performance using significantly less memory.

Problem

• Improving LoRA fine-tuning efficiency to match full training performance
• Reducing memory usage while maintaining model adaptation quality
• Developing interpretable optimization methods for low-rank adaptation

Innovation

• Alternating least squares updates optimize LoRA efficiently
• Memory-efficient optimizer matching SVD performance with less memory
• Maintains low-rank momentum estimate using LoRSum subroutine