Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning

2025-05-20

AI Summary

Vision-language models (VLMs) exhibit weak out-of-distribution generalization and over-reliance on spurious shortcuts when performing complex visual reasoning without explicit chain-of-thought (CoT) supervision. Method: We propose a reinforcement learning (RL)-driven, three-stage structured generation framework that enforces sequential output of fine-grained image descriptions, logical reasoning chains, and final answers. Inspired by GRPO, our RL framework jointly optimizes captioning, reasoning, and answering via visual question-answering fine-tuning—without requiring human-annotated CoT data. Crucially, it introduces a novel training paradigm where image descriptions serve as explicit preconditioning for subsequent reasoning. Contribution/Results: Our method significantly outperforms strong baselines—including GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—across multiple visual reasoning benchmarks. It demonstrates markedly improved robustness and generalization to unseen reasoning patterns, validating the efficacy of description-conditioned, RL-guided structured generation for enhancing deep visual understanding and reliable multimodal inference.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Learning general-purpose reasoning capabilities has long been a challenging problem in AI. Recent research in large language models (LLMs), such as DeepSeek-R1, has shown that reinforcement learning techniques like GRPO can enable pre-trained LLMs to develop reasoning capabilities using simple question-answer pairs. In this paper, we aim to train visual language models (VLMs) to perform reasoning on image data through reinforcement learning and visual question-answer pairs, without any explicit chain-of-thought (CoT) supervision. Our findings indicate that simply applying reinforcement learning to a VLM -- by prompting the model to produce a reasoning chain before providing an answer -- can lead the model to develop shortcuts from easy questions, thereby reducing its ability to generalize across unseen data distributions. We argue that the key to mitigating shortcut learning is to encourage the model to interpret images prior to reasoning. Therefore, we train the model to adhere to a caption-reason-answer output format: initially generating a detailed caption for an image, followed by constructing an extensive reasoning chain. When trained on 273K CoT-free visual question-answer pairs and using only reinforcement learning, our model, named Visionary-R1, outperforms strong multimodal models, such as GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, on multiple visual reasoning benchmarks.

Problem

• Mitigating shortcut learning in visual reasoning models
• Training VLMs without explicit chain-of-thought supervision
• Improving generalization across unseen data distributions

Innovation

• Uses reinforcement learning for visual reasoning
• Implements caption-reason-answer output format
• Trains without chain-of-thought supervision