Bayesian Fine-tuning in Projected Subspaces

2026-05-08

AI Summary

Standard LoRA lacks the ability to quantify uncertainty, often leading to overconfident and poorly calibrated models. While existing Bayesian LoRA variants introduce uncertainty modeling, they suffer from parameter inflation that causes training instability. This work proposes a novel parameter-efficient Bayesian fine-tuning framework that models weight uncertainty within an extremely low-dimensional projected subspace. It reveals for the first time that the weight covariance of large models exhibits a low-rank structure, enabling stable and efficient Bayesian optimization. By leveraging this insight, the method significantly improves model calibration and generalization with negligible parameter overhead, demonstrating the effectiveness and scalability of uncertainty modeling in low-dimensional subspaces.

Abstract

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large models by decomposing weight updates into low-rank matrices, significantly reducing storage and computational overhead. While effective, standard LoRA lacks mechanisms for uncertainty quantification, leading to overconfident and poorly calibrated models. Bayesian variants of LoRA address this limitation, but at the cost of a significantly increased number of trainable parameters, partially offsetting the original efficiency gains. Additionally, these models are harder to train and may suffer from unstable convergence. In this work, we propose a novel framework for parameter-efficient Bayesian fine-tuning, demonstrating that effective uncertainty quantification can be achieved in very low-dimensional parameter spaces. The proposed method achieves strong performance with improved calibration and generalization while maintaining computational efficiency. Our empirical findings show that, with the appropriate projection of the weight space uncertainty can be effectively modeled in a low-dimensional space, and weight covariances exhibit low ranks.

Problem

• Bayesian fine-tuning
• Low-Rank Adaptation
• uncertainty quantification
• parameter-efficient
• model calibration

Innovation

• Bayesian fine-tuning
• Low-Rank Adaptation
• Projected subspaces
• Uncertainty quantification
• Parameter-efficient learning