BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

2026-06-28

AI Summary

This work addresses the limitations of conventional Low-Rank Adaptation (LoRA), whose fixed-rank structure lacks flexibility and often leads to overconfidence and poor calibration under data scarcity. The authors propose BaRA, a Bayesian adaptive rank allocation framework that uniquely integrates Bayesian inference with dynamic rank assignment. Inspired by probabilistic topic models, BaRA employs a global-local gating mechanism to dynamically activate a sparse and disentangled subset of latent factors for each input, enabling data-driven, instance-level effective rank adaptation. Theoretical analysis reveals that the generalization error bound depends on the learned joint effective rank rather than the maximum rank, demonstrating that sparse adaptation reduces hypothesis complexity without compromising representational capacity. Experiments show that BaRA consistently outperforms standard LoRA and existing Bayesian variants across multiple natural language benchmarks, achieving notable gains in predictive accuracy, robustness, and uncertainty calibration.

Abstract

While Low-rank adaptation (LoRA) enables highly efficient fine-tuning by constraining task-specific updates to fixed low-rank subspaces, this rigid design limits representational flexibility and often results in overconfident predictions and miscalibrated uncertainty, especially in low-data regimes. Recent Bayesian LoRA variants improve uncertainty estimation by modeling posterior distributions over adaptation parameters. However, these approaches typically rely on fixed or heuristically determined ranks, overlooking the inherently context-dependent nature of adaptation capacity. In this paper, we propose BaRA, a Bayesian Adaptive Rank Allocation framework for parameter-efficient fine-tuning. Drawing inspiration from probabilistic topic models, BaRA dynamically allocates adaptation capacity by activating a sparse, context-dependent subset of disentangled latent factors, enabling instance-wise variation in effective rank. This Bayesian formulation provides principled, data-driven capacity control, mitigating over-parameterization while preserving expressiveness. Beyond the modeling contribution, we provide a complexity-theoretic generalization analysis showing that the generalization gap of BaRA depends on the learned joint effective rank $\bar{s}_{Φ,θ}$ induced by the global-local gate, rather than the maximum rank $r$. This result explains why sparse adaptive rank allocation can reduce the effective hypothesis complexity while preserving input-dependent expressiveness. Extensive experiments on diverse natural language benchmarks demonstrate that BaRA consistently improves predictive performance, robustness, and uncertainty calibration compared to standard LoRA and existing Bayesian LoRA variants.

Problem

• Low-rank adaptation
• Bayesian fine-tuning
• Rank allocation
• Uncertainty calibration
• Parameter-efficient tuning

Innovation

• Bayesian adaptive rank allocation
• parameter-efficient fine-tuning
• low-rank adaptation
• uncertainty calibration
• dynamic rank selection