On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference

2026-05-28

AI Summary

This work addresses the limitations of existing LoRA fine-tuning methods in effectively capturing epistemic uncertainty and the suboptimal performance of discrete multimodal strategies—such as deep ensembles—that overlook continuous low-loss pathways connecting distinct optima in parameter space. To overcome these issues, the authors propose LoRA-Curve, which, for the first time, constructs piecewise Bézier curves within the LoRA parameter space to link independently optimized solutions, revealing the existence of continuous low-loss valleys traversable via multi-segment curves. By incorporating Jensen–Shannon divergence regularization and flat-minimum perturbations, the method enhances predictive diversity and uncertainty estimation. Experiments on the Qwen2.5-7B model demonstrate that traversing these low-loss valleys improves mutual information in Bayesian model averaging while preserving task performance across reasoning and classification benchmarks.

Abstract

While parameter-efficient fine-tuning methods like low-rank adaptation (LoRA) are standard for large language models, principled estimation of epistemic uncertainty remains challenging. Recent results in the LoRA regime suggest that discrete multi-mode approaches such as deep ensembles offer little benefit over single-mode methods. This contradicts broader observations in deep learning, where ensembling independent optima typically improves generalization, and linking these modes through continuous low-loss valleys further enhances Bayesian model averaging (BMA). Whether such structure exists in the LoRA space and whether it yields functional diversity missed by local or discrete methods has not been studied. We introduce LoRA-Curve, a segmented Bézier curve parameterization in the LoRA space, with two variants: a free configuration that jointly optimizes all control points, and an anchored configuration that connects independently fine-tuned LoRA optima. We prove pathwise continuity and Lipschitz regularity of the loss along the curve and empirically show, across reasoning and classification benchmarks with Qwen2.5 7B, that linear interpolation encounters loss barriers, while our anchored multi-segment curves connect independent optima through continuous low-loss valleys. Combined with flat-minima perturbations and a Jensen-Shannon divergence regularizer, LoRA-Curve yields measurably higher mutual information of the predictive distribution without sacrificing performance, and links continuous parameter-space traversal to functional diversity.

Problem

• LoRA
• Bayesian inference
• epistemic uncertainty
• low-loss valleys
• functional diversity

Innovation

• LoRA-Curve
• low-loss valleys
• Bayesian inference
• functional diversity
• parameter-efficient fine-tuning