Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

2026-06-24

AI Summary

This work addresses the tendency of multimodal large language models (MLLMs) to generate hallucinations that contradict visual inputs due to overreliance on linguistic priors. To mitigate this, the authors propose VIGIL, a reinforcement learning–based post-training framework that penalizes “blind confidence” under counterfactual blind-spot conditions, thereby enforcing causal alignment between model outputs and visual content. VIGIL incorporates geometric constraints to maximize visual–language mutual information and achieves state-of-the-art performance using only 25% of the typical preference data. Notably, it demonstrates emergent spatial localization capabilities without explicit supervision. Extensive experiments show that VIGIL significantly outperforms existing alignment methods across multiple hallucination and reasoning benchmarks while preserving strong performance on pure-text tasks, confirming its effectiveness and generalization ability.

Abstract

Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception, enabling joint reasoning over images and text. Despite inheriting strong reasoning capabilities from LLMs, they remain prone to hallucinations that contradict their visual inputs. Mechanistic studies indicate that this weakness stems from visual laziness: MLLMs encode the correct visual evidence internally, but overly rely on strong language priors during response. Existing alignment methods, such as direct preference optimization, primarily optimize outcome-level rewards based on text. This introduces an optimization bias toward linguistic shortcuts, leading to responses that often contradict the visual evidence. To address this, we propose Visual Information Gain In aLignment (VIGIL), a reinforcement-learning (RL) post-training framework that shifts the focus from numerical reward fitting to causal visual grounding. VIGIL introduces a geometric constraint that explicitly maximizes the mutual information between the visual input and the generated response. We achieve this by penalizing "blind confidence" instances where the model remains improperly certain even when textual-visual attention is masked to create a counterfactual blind state. Extensive experiments show that VIGIL consistently outperforms recent alignment methods across hallucination and reasoning benchmarks without compromising text-only capabilities. Our approach matches the full-data performance of state-of-the-art methods using only 25% of the preference data and even demonstrates emergent spatial grounding capabilities without explicit bounding box supervision.

Problem

• visual hallucination
• visual laziness
• multimodal alignment
• counterfactual reasoning
• visual grounding

Innovation

• visual laziness
• counterfactual visual alignment
• mutual information maximization
• reinforcement learning post-training
• multimodal hallucination mitigation