Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis

2025-05-15

AI Summary

This paper addresses the evaluation challenge of “implicit visual miscomprehension” (IVM) in multimodal large language models (MLLMs)—a phenomenon where models produce correct answers without genuinely understanding visual inputs. Methodologically, we decouple visual and textual pathways within causal attention mechanisms and analyze cross-layer attention distribution patterns; we propose a scale-invariant metric, “attention accuracy,” and introduce the first dedicated IVM benchmark. Our contributions are threefold: (1) we eliminate positional bias for the first time, enabling fine-grained diagnostic analysis and single-modality generalization; (2) we empirically uncover a mechanism wherein deeper-layer attention progressively converges onto image regions semantically aligned with the answer; and (3) our approach significantly enhances the reliability of visual understanding assessment, demonstrating strong generalization across both multimodal and unimodal tasks.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advancements have enhanced the capability of Multimodal Large Language Models (MLLMs) to comprehend multi-image information. However, existing benchmarks primarily evaluate answer correctness, overlooking whether models genuinely comprehend the visual input. To address this, we define implicit visual misunderstanding (IVM), where MLLMs provide correct answers without fully comprehending the visual input. Through our analysis, we decouple the visual and textual modalities within the causal attention module, revealing that attention distribution increasingly converges on the image associated with the correct answer as the network layers deepen. This insight leads to the introduction of a scale-agnostic metric, extit{attention accuracy}, and a novel benchmark for quantifying IVMs. Attention accuracy directly evaluates the model's visual understanding via internal mechanisms, remaining robust to positional biases for more reliable assessments. Furthermore, we extend our approach to finer granularities and demonstrate its effectiveness in unimodal scenarios, underscoring its versatility and generalizability.

Problem

• Identifying implicit visual misunderstandings in MLLMs
• Evaluating visual comprehension beyond answer correctness
• Proposing attention accuracy for robust visual understanding assessment

Innovation

• Decouple visual and textual modalities in attention
• Introduce scale-agnostic attention accuracy metric
• Extend approach to finer granularities effectively