Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding

2026-03-09

AI Summary

This work addresses the susceptibility of multimodal large reasoning models (MLRMs) to hallucination in visual question answering, particularly under high-entropy conditions such as transitional phrases. To mitigate this, the authors propose LEAD, a plug-and-play latent entropy-aware decoding strategy. During high-entropy decoding stages, LEAD performs latent semantic fusion via probability-weighted continuous embeddings and enhances image grounding through prior-guided visual anchors; during low-entropy stages, it reverts to conventional discrete token generation. LEAD is the first approach to integrate latent hyper-positional representations with entropy-aware mechanisms, enabling dense contextual reasoning under high uncertainty and reducing overreliance on discrete textual outputs. Experiments demonstrate that LEAD significantly lowers hallucination rates and improves both reasoning reliability and accuracy across multiple benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advancements in multimodal large reasoning models (MLRMs) have significantly improved performance in visual question answering. However, we observe that transition words (e.g., because, however, and wait) are closely associated with hallucinations and tend to exhibit high-entropy states. We argue that adequate contextual reasoning information can be directly extracted from the token probability distribution. Inspired by superposed representation theory, we propose leveraging latent superposed reasoning to integrate multiple candidate semantics and maintain latent reasoning trajectories. The hypothesis is that reliance on discrete textual inputs may drive the model toward sequential explicit reasoning, underutilizing dense contextual cues during high-entropy reasoning stages. Therefore, we propose constructing rich semantic representations from the token probability distributions to enhance in-context reasoning. With this goal, we present Latent Entropy-Aware Decoding (LEAD), an efficient plug-and-play decoding strategy that leverages semantic context to achieve reliable reasoning. The heart of our method lies in entropy-aware reasoning mode switching. The model employs probability-weighted continuous embeddings under high-entropy states and transitions back to discrete token embeddings as entropy decreases. Moreover, we propose a prior-guided visual anchor injection strategy that encourages the model to focus on visual information. Extensive experiments show that LEAD effectively mitigates hallucinations across various MLRMs on multiple benchmarks.

Problem

• hallucinations
• multimodal large reasoning models
• visual question answering
• high-entropy states
• contextual reasoning

Innovation

• Latent Entropy-Aware Decoding
• hallucination mitigation
• multimodal reasoning
• entropy-aware switching
• visual anchor injection