CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation

2026-06-30

AI Summary

This work addresses the challenge of quantifying uncertainty in open-domain multimodal large language models, where responses are inherently open-ended and sources of uncertainty are complex. The authors propose the first structured disentanglement framework that decomposes uncertainty into two interpretable components: a context-dependent term capturing ambiguity induced by the task or prompt, and a multiplicity-related term measuring the number of plausible answers compatible with the input. A lightweight post-processing module efficiently estimates both components without requiring autoregressive generation or multiple sampling passes. Evaluated across diverse multimodal benchmarks—including hallucination detection and visual question answering—the method significantly outperforms existing baselines while maintaining high computational efficiency.

Abstract

Uncertainty estimation has been a long-standing challenge in AI models; it amounts to "knowing what you don't know," and metacognition is notoriously difficult even for humans (cf. the Dunning-Kruger effect). Although it is still far from solved even in simpler classification systems, tackling it in multimodal large language models (MLLMs) is becoming increasingly important. Within MLLMs, uncertainty can stem from any of the diverse sources as well as from their relationships, and further can stem from the unbounded answers in the open-ended setting. To tackle the issues, we propose CoMet, an MLLM uncertainty estimation method by decomposing uncertainty into a context-specific term and a multiplicity-specific term. The former captures ambiguity induced by the given context (e.g., task or prompt), while the latter captures how many plausible answers determined by the context remain compatible with the given input. We train a lightweight post-hoc uncertainty module to estimate these quantities, which enables efficient uncertainty estimation without autoregressive answer generation or repeated sampling. Experiments on various open-ended multimodal benchmarks, hallucination detection, and multiple-choice visual question answering benchmarks show that CoMet consistently improves uncertainty estimation over existing baselines while remaining efficient in practice. Code is available at https://github.com/princetonvisualai/comet_uncertainty

Problem

• uncertainty estimation
• multimodal large language models
• open-ended generation
• hallucination detection

Innovation

• uncertainty decomposition
• multimodal large language models
• context-specific uncertainty
• multiplicity-specific uncertainty
• post-hoc uncertainty estimation