Visually-Guided Policy Optimization for Multimodal Reasoning

2026-04-10

AI Summary

This work addresses the tendency of vision-language models in reinforcement learning to overlook visual information due to textual dominance and suffer from temporal visual forgetting during reasoning. To mitigate these issues, the authors propose the Vision-Guided Policy Optimization (VGPO) framework, which introduces a novel visual attention compensation mechanism and a progressive visual expectation enhancement strategy. These components are integrated with intra- and inter-trajectory dual-granularity advantage reweighting to significantly strengthen the model’s focus on and fidelity to visual cues. Experimental results demonstrate that VGPO substantially increases visual activation levels and improves overall performance on multimodal mathematical reasoning and vision-dependent tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning ability of vision-language models (VLMs). However, the inherent text-dominated nature of VLMs often leads to insufficient visual faithfulness, characterized by sparse attention activation to visual tokens. More importantly, our empirical analysis reveals that temporal visual forgetting along reasoning steps exacerbates this deficiency. To bridge this gap, we propose Visually-Guided Policy Optimization (VGPO), a novel framework to reinforce visual focus during policy optimization. Specifically, VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues, while progressively elevating visual expectations in later steps to counteract visual forgetting. Building on this mechanism, we implement a dual-grained advantage re-weighting strategy: the intra-trajectory level highlights tokens exhibiting relatively high visual activation, while the inter-trajectory level prioritizes trajectories demonstrating superior visual accumulation. Extensive experiments demonstrate that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks.

Problem

• visual faithfulness
• temporal visual forgetting
• vision-language models
• multimodal reasoning
• visual attention

Innovation

• Visually-Guided Policy Optimization
• Visual Attention Compensation
• Temporal Visual Forgetting
• Dual-grained Advantage Reweighting
• Vision-Language Models