Self-Rewarding Vision-Language Model via Reasoning Decomposition

2025-08-27

AI Summary

Vision-language models (VLMs) suffer from two persistent issues: visual hallucination—generating descriptions of non-existent image content—and language shortcuts—bypassing visual input in favor of textual priors. These stem from post-training paradigms that supervise only final outputs, lacking explicit guidance for intermediate visual reasoning. To address this, we propose Vision-SR1, a self-rewarding framework requiring no external annotations or reward models. Its core innovation is decoupling reasoning into two stages—visual perception and linguistic inference—and leveraging the model’s own perceptual outputs to perform self-re-evaluation, thereby generating intrinsic reinforcement signals. Vision-SR1 jointly optimizes both the self-reward signal and standard output supervision, enabling co-adaptation of visual and linguistic capabilities. Experiments demonstrate that Vision-SR1 significantly mitigates visual hallucination, reduces reliance on language priors, and improves robustness and generalization across diverse multimodal understanding tasks.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Vision-Language Models (VLMs) often suffer from visual hallucinations, saying things that are not actually in the image, and language shortcuts, where they skip the visual part and just rely on text priors. These issues arise because most post-training methods for VLMs rely on simple verifiable answer matching and supervise only final outputs, leaving intermediate visual reasoning without explicit guidance. As a result, VLMs receive sparse visual signals and often learn to prioritize language-based reasoning over visual perception. To mitigate this, some existing methods add visual supervision using human annotations or distilled labels from external large models. However, human annotations are labor-intensive and costly, and because external signals cannot adapt to the evolving policy, they cause distributional shifts that can lead to reward hacking. In this paper, we introduce Vision-SR1, a self-rewarding method that improves visual reasoning without relying on external visual supervisions via reinforcement learning. Vision-SR1 decomposes VLM reasoning into two stages: visual perception and language reasoning. The model is first prompted to produce self-contained visual perceptions that are sufficient to answer the question without referring back the input image. To validate this self-containment, the same VLM model is then re-prompted to perform language reasoning using only the generated perception as input to compute reward. This self-reward is combined with supervision on final outputs, providing a balanced training signal that strengthens both visual perception and language reasoning. Our experiments demonstrate that Vision-SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts across diverse vision-language tasks.

Problem

• Mitigates visual hallucinations in vision-language models
• Reduces language shortcuts by improving visual reasoning
• Eliminates need for external visual supervision signals

Innovation

• Self-rewarding reinforcement learning without external supervision
• Decomposes reasoning into visual perception and language stages
• Uses self-generated perceptions to compute internal rewards