C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models

2025-05-23

AI Summary

LoRA-based few-shot fine-tuning often yields overconfident predictions, and existing uncertainty-aware approaches neglect the influence of input features on uncertainty estimation. To address this, we propose Context-Aware Lightweight LoRA (CALoRA), the first method to explicitly incorporate input features into the posterior distribution over low-rank adaptation parameters. CALoRA integrates Bayesian low-rank decomposition, context-conditioned parameter generation, and a lightweight neural modulator to enable sample-wise adaptive uncertainty calibration. Extensive experiments across diverse few-shot benchmarks demonstrate that CALoRA significantly outperforms prior uncertainty-aware LoRA methods, achieving superior calibration, generalization, and robustness. Ablation studies confirm that the context-aware module is pivotal to these gains.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs), but it often produces overconfident predictions in data-scarce few-shot settings. To address this issue, several classical statistical learning approaches have been repurposed for scalable uncertainty-aware LoRA fine-tuning. However, these approaches neglect how input characteristics affect the predictive uncertainty estimates. To address this limitation, we propose Contextual Low-Rank Adaptation ( extbf{C-LoRA}) as a novel uncertainty-aware and parameter efficient fine-tuning approach, by developing new lightweight LoRA modules contextualized to each input data sample to dynamically adapt uncertainty estimates. Incorporating data-driven contexts into the parameter posteriors, C-LoRA mitigates overfitting, achieves well-calibrated uncertainties, and yields robust predictions. Extensive experiments demonstrate that C-LoRA consistently outperforms the state-of-the-art uncertainty-aware LoRA methods in both uncertainty quantification and model generalization. Ablation studies further confirm the critical role of our contextual modules in capturing sample-specific uncertainties. C-LoRA sets a new standard for robust, uncertainty-aware LLM fine-tuning in few-shot regimes.

Problem

• Overconfident predictions in few-shot LoRA fine-tuning
• Neglecting input impact on uncertainty estimates
• Need for robust uncertainty-aware LLM adaptation

Innovation

• Dynamic lightweight LoRA modules per input
• Data-driven contexts in parameter posteriors
• Sample-specific uncertainty capture via contextual modules