Uncertainty-Aware Decision Making in Multimodal Large Language Models

2026-08-17

AI Summary

本文针对多模态大语言模型在处理跨模态信息时可能产生的不确定性问题，提出了一种决策为中心的框架来校准和控制这些不确定性，并通过多种方法改善模型在面对不充分、冲突或高风险证据时的行为。

Abstract

Multimodal large language models (MLLMs) increasingly answer questions whose correctness depends on visual, textual, temporal, acoustic, document, chart, or embodied evidence. Their failures are therefore not only linguistic. A fluent answer may conceal poor input quality, a perceptual error, weak grounding, conflict between modalities, unstable reasoning, distribution shift, or a question that is not answerable from the supplied evidence. This survey organizes the literature on uncertainty-aware MLLMs around a decision-centered framework: uncertainty sources give rise to observable signals, signals must be calibrated or controlled for risk, and calibrated uncertainty should determine the system action. We review work on token and logit uncertainty, semantic disagreement, perturbation instability, grounding and attribution scores, verbalized confidence, verifier and judge scores, conformal prediction, selective answering, abstention, clarification, retrieval, self-checking, and escalation. The central argument is that uncertainty should not be evaluated only as a confidence number; it should be evaluated by whether it improves behavior under insufficient, conflicting, shifted, or high-risk multimodal evidence. We position this survey against text-only uncertainty and abstention surveys, broad MLLM surveys, MLLM hallucination surveys, and safety-oriented reviews. We conclude with open problems in source-aware decomposition, action-aware benchmarks, calibration under shift, black-box uncertainty estimation, broader modality coverage, reproducible reporting, and human-centered uncertainty communication.

Problem

• Multimodal Large Language Models
• Uncertainty
• Decision Making
• Evidence
• Risk

Innovation

• uncertainty-aware
• multimodal large language models
• decision-centered framework
• calibrated uncertainty
• system action