Grounded Reinforcement Learning for Visual Reasoning

2025-05-29

AI Summary

In visual reasoning, abstract inference and spatial perception are often decoupled, and existing RL models struggle to jointly model visual attention, perceptual interpretation, and spatial evidence. To address this, we propose ViGoRL—a novel reinforcement learning framework that explicitly anchors each reasoning step to image-space coordinates. ViGoRL employs multi-round PPO optimization to dynamically scale coordinates and generate heatmap-guided, fine-grained visual feedback. It integrates multi-scale cropping-and-recoding, spatial coordinate regression, and human-preference-aligned interpretability evaluation. On the V*Bench visual search benchmark, ViGoRL achieves 86.4% accuracy, substantially outperforming supervised fine-tuning and non-grounding RL baselines. It also sets new state-of-the-art results across diverse benchmarks—including SAT-2, BLINK, ScreenSpot, and VisualWebArena. Human evaluation confirms its superior spatial localization precision and clear, step-wise reasoning interpretability.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Search and Optimization: Learning to Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

While reinforcement learning (RL) over chains of thought has significantly advanced language models in tasks such as mathematics and coding, visual reasoning introduces added complexity by requiring models to direct visual attention, interpret perceptual inputs, and ground abstract reasoning in spatial evidence. We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with RL to explicitly anchor each reasoning step to specific visual coordinates. Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces, guiding visual attention to task-relevant regions at each step. When fine-grained exploration is required, our novel multi-turn RL framework enables the model to dynamically zoom into predicted coordinates as reasoning unfolds. Across a diverse set of visual reasoning benchmarks--including SAT-2 and BLINK for spatial reasoning, V*bench for visual search, and ScreenSpot and VisualWebArena for web-based grounding--ViGoRL consistently outperforms both supervised fine-tuning and conventional RL baselines that lack explicit grounding mechanisms. Incorporating multi-turn RL with zoomed-in visual feedback significantly improves ViGoRL's performance on localizing small GUI elements and visual search, achieving 86.4% on V*Bench. Additionally, we find that grounding amplifies other visual behaviors such as region exploration, grounded subgoal setting, and visual verification. Finally, human evaluations show that the model's visual references are not only spatially accurate but also helpful for understanding model reasoning steps. Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.

Problem

• Anchors reasoning steps to visual coordinates for accuracy
• Improves visual attention in complex reasoning tasks
• Enhances performance in diverse visual reasoning benchmarks

Innovation

• Uses RL to anchor reasoning to visual coordinates
• Multi-turn RL enables dynamic zoom for exploration
• Spatially grounded reasoning improves visual attention