Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning

2026-07-02

AI Summary

Existing large vision-language models struggle to effectively leverage visual inputs during self-reflection, limiting their ability to correct errors on out-of-distribution images. This work proposes VRRL, a reinforcement learning framework that, for the first time, integrates experience replay with random masking of trajectory prefixes to guide the model toward greater reliance on visual information for accurate error correction during training. By replaying diverse failure states and optimizing policies with explicit visual grounding, VRRL significantly improves out-of-distribution accuracy on tasks such as table and chart understanding and spatial navigation, outperforming both standard reinforcement learning approaches and conventional reflection-based fine-tuning methods.

Abstract

Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images. To address this issue, we propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection. First, we randomly mask trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions rather than making early mistakes. Second, we introduce buffered roll-ins from an experience replay buffer to expose the model to diverse failure states that it must learn to correct. We evaluate our approach on visual grounding tasks involving tables and charts, as well as spatial navigation benchmarks. While off-the-shelf and conventionally fine-tuned models degrade substantially under distribution shift, our method substantially improves average out-of-distribution accuracy over standard RL and reflection-oriented fine-tuning baselines by using self-reflection effectively.

Problem

• visually grounded
• self-reflection
• vision-language models
• out-of-distribution
• visual grounding

Innovation

• visually grounded self-reflection
• reinforcement learning
• experience replay
• chain of thought
• out-of-distribution generalization