Information-Regularized Attention for Visual-Centric Reasoning

2026-07-01

AI Summary

This work addresses object hallucination, weak visual grounding, and catastrophic forgetting in vision-language models during instruction tuning, which stem from the standard next-token prediction objective’s lack of explicit constraints on visual representation learning. To mitigate these issues, the authors propose Information-Regularized Attention (IRA), a novel mechanism that introduces stochastic attention into intermediate Transformer layers—not merely as a regularizer but as a core component for modulating visual information. Through local reparameterization, IRA transforms visual uncertainty into data-independent local noise, effectively suppressing attention collapse and yielding smoother embedding curvature trajectories. This approach significantly alleviates the aforementioned challenges, enhancing both visual grounding and training stability.

Abstract

Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning. We claim these failures result from a lack of explicit control over visual representation learning during the standard next-token prediction objective. As a result, visual embeddings thus become passively optimized and prone to injecting redundant or spurious signals. To counter this, we introduce Information-Regularized Attention (IRA), a stochastic attention mechanism that explicitly regulates the amount of visual information injected into the hidden states of intermediate transformer layers. This local reparameterization translates uncertainty about visual representations into local noise that is independent across data points. Beyond evaluating model performance, we also quantify embedding properties, where IRA produces smoother curvature trajectories and suppresses attention-sink across all layers, indicating a more stable transformation of the visual signal. Our results suggest that stochastic attention is not merely a regularizer but a key contributor to representation learning in a generative architecture, offering a new direction for building more reliable VLMs.

Problem

• object hallucination
• visual grounding
• catastrophic forgetting
• visual representation learning
• vision-language models

Innovation

• Information-Regularized Attention
• stochastic attention
• visual grounding
• vision-language models
• representation learning