Bernoulli-LoRA: A Theoretical Framework for Randomized Low-Rank Adaptation

2025-08-05

AI Summary

Existing parameter-efficient fine-tuning (PEFT) methods—such as Low-Rank Adaptation (LoRA)—lack a unified theoretical framework and suffer from limited convergence analysis. Method: We propose Bernoulli-LoRA, the first LoRA-based framework that employs Bernoulli random sampling to construct a unified low-rank update mechanism, inherently accommodating diverse LoRA variants. Integrating non-convex and convex optimization theory, it establishes universal convergence guarantees for stochastic optimizers—including SGD, PAGE, MVR, and MARINA—under non-smooth objectives and adaptive step sizes. Results: Our theoretical analysis covers broad classes of objective functions, while empirical evaluation across multiple downstream tasks demonstrates both rigorous theoretical foundations and superior practical performance. This work introduces the first unified convergence analysis paradigm for PEFT grounded in randomized structural design.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large foundational models to specific tasks, particularly as model sizes continue to grow exponentially. Among PEFT methods, Low-Rank Adaptation (LoRA) (arXiv:2106.09685) stands out for its effectiveness and simplicity, expressing adaptations as a product of two low-rank matrices. While extensive empirical studies demonstrate LoRA's practical utility, theoretical understanding of such methods remains limited. Recent work on RAC-LoRA (arXiv:2410.08305) took initial steps toward rigorous analysis. In this work, we introduce Bernoulli-LoRA, a novel theoretical framework that unifies and extends existing LoRA approaches. Our method introduces a probabilistic Bernoulli mechanism for selecting which matrix to update. This approach encompasses and generalizes various existing update strategies while maintaining theoretical tractability. Under standard assumptions from non-convex optimization literature, we analyze several variants of our framework: Bernoulli-LoRA-GD, Bernoulli-LoRA-SGD, Bernoulli-LoRA-PAGE, Bernoulli-LoRA-MVR, Bernoulli-LoRA-QGD, Bernoulli-LoRA-MARINA, and Bernoulli-LoRA-EF21, establishing convergence guarantees for each variant. Additionally, we extend our analysis to convex non-smooth functions, providing convergence rates for both constant and adaptive (Polyak-type) stepsizes. Through extensive experiments on various tasks, we validate our theoretical findings and demonstrate the practical efficacy of our approach. This work is a step toward developing theoretically grounded yet practically effective PEFT methods.

Problem

• Theoretical understanding of LoRA methods is limited
• Unify and extend existing LoRA approaches theoretically
• Analyze convergence guarantees for various Bernoulli-LoRA variants

Innovation

• Probabilistic Bernoulli mechanism for matrix selection
• Unifies and extends existing LoRA approaches
• Convergence guarantees for multiple algorithm variants