Improving Audio Question Answering with Variational Inference

2026-01-19

AI Summary

This work addresses the issue of miscalibration and overconfidence in multimodal large language models (MLLMs) for audio question answering, which undermines their reliability in risk-sensitive applications. To mitigate this, the study introduces the Improved Variational Online Newton (IVON) optimizer into the fine-tuning process—an approach that explicitly models parameter uncertainty through variational inference. By incorporating uncertainty-aware optimization, the proposed method not only enhances predictive accuracy but also significantly improves confidence calibration, thereby alleviating overconfidence. This advancement yields more trustworthy outputs, making the model better suited for high-stakes scenarios such as selective prediction, where calibrated confidence estimates are critical for decision-making.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Variational inference (VI) provides a principled framework for estimating posterior distributions over model parameters, enabling explicit modeling of weight uncertainty during optimization. By capturing this uncertainty, VI improves the reliability of predictions, yielding better calibrated outputs. In this work, we investigate the benefits of VI for challenging multimodal understanding and reasoning by applying the Improved Variational Online Newton (IVON), a recent VI optimizer, to fine-tuning a multimodal large language model on audio question answering tasks. Our results show that VI not only enhances predictive accuracy but also significantly improves calibration, reducing the model's overconfidence. These advances further support risk-sensitive applications such as selective prediction, where reliable confidence estimates are crucial.

Problem

• Audio Question Answering
• Multimodal Understanding
• Model Calibration
• Overconfidence
• Uncertainty Estimation

Innovation

• Variational Inference
• Audio Question Answering
• Multimodal Large Language Model
• Model Calibration
• IVON