Improving Reasoning in Vision-Language Models via Perception Verified Self-Training

2026-06-20

AI Summary

This work addresses the susceptibility of current vision-language models to visual hallucinations and linguistic shortcuts, which often result in reasoning without reliable perceptual grounding. To mitigate these issues, the authors propose a perception-verification-based self-training framework that decouples perception from reasoning. The approach introduces an unsupervised caption evaluation mechanism (PerceptEval) and a “caption–reasoning–conclusion” template to triage training data according to perceptual quality and answer correctness. A two-stage curriculum learning strategy is designed to progressively refine model behavior. By integrating chain-of-thought reasoning, vision–text alignment assessment, and conditional generation—without requiring additional annotations—the method significantly enhances visual grounding and complex reasoning accuracy, effectively reducing both hallucination and shortcut reliance.

Abstract

Achieving human-like reasoning in Vision-Language Models (VLMs) remains a long-standing challenge. Recent approaches leverage Chain-of-Thought (CoT) rationales generated by human annotators or proprietary models to improve reasoning, which is costly and difficult to scale. Self-training offers a promising alternative by using models own outputs as supervision. However, existing methods often suffer from visual hallucinations -- where rationales describe non-existent visual content, and language shortcuts -- where predictions rely on textual priors rather than true visual grounding, as rationales are typically filtered only by answer correctness without verifying visual perception. To address this limitation, we propose a perception-verified self-training framework that enforces visually grounded reasoning. First, our method employs a CoT template (caption-reasoning-conclusion) that disentangles perception from reasoning, enabling independent verification of visual understanding. To compensate for the absence of ground-truth captions, we propose PerceptEval, an unsupervised method that evaluates caption quality based on its alignment with visual and textual elements present in the image. Using caption verification together with answer correctness, we partition the data into three subsets: easy (correct caption and conclusion), medium (correct caption but incorrect conclusion), and hard (incorrect caption). Building on this partitioning, we design a two-stage curriculum learning strategy. In Stage 1, the model is trained on easy examples and subsequently in Stage 2, medium samples are incorporated through a caption-guided reasoning enhancement procedure that regenerates reasoning conditioned on verified captions. Only regenerated samples with the correct conclusions are retained.

Problem

• visual hallucination
• language shortcuts
• vision-language reasoning
• perception verification
• self-training

Innovation

• perception-verified self-training
• visual grounding
• Chain-of-Thought reasoning
• PerceptEval
• curriculum learning