Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

2026-06-25

AI Summary

This work addresses the visual under-conditioning problem in existing self-evolving multimodal large language models, where decoders overly rely on linguistic priors and neglect visual inputs during unsupervised training, thereby limiting performance in image-text generation and visual question answering. To mitigate this, the authors propose VISE, a novel framework that introduces invariance-based visual conditioning regularization within the self-evolution paradigm. Specifically, VISE employs geometric invariance rewards—enforcing output consistency under spatial transformations—and semantic invariance rewards—penalizing generations lacking visual grounding—to directly regularize the decoder’s attention to visual tokens. Notably, VISE requires no external models, annotations, or multi-agent setups, operating solely on unlabeled images. Evaluated on Qwen3-VL-2B, it improves CIDEr scores by 16.85 and 19.66 on COCO and TextCaps, respectively, reduces object hallucination by 5.0 Chair-I points, and demonstrates strong generalization across 18 benchmarks spanning diverse model families and scales.

Abstract

Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision--language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18 benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE

Problem

• visual under-conditioning
• large multimodal models
• visual reasoning
• self-evolving
• visual tokens

Innovation

• visual invariance
• self-evolving LMMs
• unsupervised learning
• visual under-conditioning
• invariance-based rewards