Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models

2026-06-20

AI Summary

This work addresses the absence of standardized benchmarks for evaluating Bayesian low-rank adaptation methods in multimodal language models with respect to uncertainty calibration, robustness under distribution shift, and active learning performance. To bridge this gap, we introduce Bayesian Adaptation Gym (BAG), the first evaluation framework specifically designed for Bayesian low-rank adaptation in large multimodal models. BAG integrates both classical and state-of-the-art approaches across three core evaluation dimensions and provides a modular, extensible open-source benchmarking platform. Through extensive experiments, we systematically analyze how different adaptation strategies vary in effectiveness across model scales and task types, thereby establishing a reliable foundation and infrastructure for the evaluation of efficient Bayesian fine-tuning in multimodal settings.

Abstract

Large multi-modal language models are increasingly deployed in high-stakes domains, making well-calibrated uncertainty essential. Traditional Bayesian methods approximate posteriors over all model weights, which becomes intractable for modern large models. For this reason, recent work instead considers Bayesian low-rank adaptation to enable tractable posterior approximation. Due to a lack of a standardized benchmark to evaluate these approaches, it remains unclear where these methods provide meaningful benefits. To fill this gap, we introduce Bayesian Adaptation Gym (BAG), a benchmark for the Bayesian adaptation of multi-modal language models. BAG provides reference implementations of classic Bayesian baselines and state-of-the-art adaptation methods, along with a multi-modal dataset and task suite designed to probe calibration, robustness under distribution shift, and decision-making under uncertainty via active learning. Using BAG, we conduct and report extensive experiments across model sizes, datasets, and tasks to highlight the successes and failures of current Bayesian adaptation approaches. To enable further research, BAG is fully open source: https://github.com/SRI-CSL/BayesAdapt.

Problem

• Bayesian adaptation
• low-rank adaptation
• multi-modal language models
• uncertainty calibration
• benchmark

Innovation

• Bayesian low-rank adaptation
• multi-modal language models
• uncertainty calibration
• benchmark
• active learning