Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs

2025-10-01

AI Summary

MLLMs exhibit limited performance in fine-grained visual perception—such as detecting small objects in high-resolution images or localizing critical moments in long videos—due to inherent resolution and temporal modeling constraints. Existing approaches rely on task-specific fine-tuning, resulting in poor generalizability and high computational complexity. This paper proposes a training-free, unified framework that leverages the intrinsic uncertainty of MLLM outputs—quantified via entropy—as an active guidance signal to dynamically select salient visual regions or temporal segments. The method adaptively focuses on discriminative content without modifying model architecture or parameters, requiring only black-box inference from off-the-shelf MLLMs. Evaluated on visual search, long-video understanding, and temporal grounding, it matches or exceeds specialized fine-tuned methods while achieving superior cross-task generalization and deployment efficiency. Our work establishes a lightweight, general-purpose paradigm for fine-grained multimodal understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multimodal Large Language Models (MLLMs) often struggle with fine-grained perception, such as identifying small objects in high-resolution images or finding key moments in long videos. Existing works typically rely on complicated, task-specific fine-tuning, which limits their generalizability and increases model complexity. In this work, we propose an effective, training-free framework that uses an MLLM's intrinsic uncertainty as a proactive guidance signal. Our core insight is that a model's output entropy decreases when presented with relevant visual information. We introduce a unified mechanism that scores candidate visual inputs by response uncertainty, enabling the model to autonomously focus on the most salient data. We apply this simple principle to three complex visual tasks: Visual Search, Long Video Understanding, and Temporal Grounding, allowing off-the-shelf MLLMs to achieve performance competitive with specialized, fine-tuned methods. Our work validates that harnessing intrinsic uncertainty is a powerful, general strategy for enhancing fine-grained multimodal performance.

Problem

• Enhancing fine-grained perception in multimodal models
• Reducing reliance on task-specific fine-tuning complexity
• Using intrinsic uncertainty to guide visual input selection

Innovation

• Training-free framework using MLLM intrinsic uncertainty
• Unified mechanism scores inputs by response uncertainty
• Autonomously focuses on salient data for complex tasks