Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning

2025-06-04

AI Summary

Multimodal large language models (MLLMs) exhibit insufficient reasoning capabilities for complex mathematical and logical visual reasoning, and direct application of reinforcement learning (RL) fails to activate deep, multi-step cross-modal reasoning. Method: We propose a staged RL framework: (1) cold-start initialization using curated pure-text data—first demonstrating its superiority over most multimodal reasoning models; (2) enhancement of the GRPO algorithm to mitigate gradient stagnation in multimodal settings; and (3) a perception-anchored, cognition-reasoning co-evolutionary three-stage training paradigm (multimodal RL → text-only RL fine-tuning) with perception–cognition decoupling. Contribution/Results: Our ReVisual-R1 model establishes new open-source SOTA among 7B MLLMs on MathVerse, MathVision, LogicVista, and AIME2024/2025 benchmarks, achieving significant gains in multi-step cross-modal reasoning performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Inspired by the remarkable reasoning capabilities of Deepseek-R1 in complex textual tasks, many works attempt to incentivize similar capabilities in Multimodal Large Language Models (MLLMs) by directly applying reinforcement learning (RL). However, they still struggle to activate complex reasoning. In this paper, rather than examining multimodal RL in isolation, we delve into current training pipelines and identify three crucial phenomena: 1) Effective cold start initialization is critical for enhancing MLLM reasoning. Intriguingly, we find that initializing with carefully selected text data alone can lead to performance surpassing many recent multimodal reasoning models, even before multimodal RL. 2) Standard GRPO applied to multimodal RL suffers from gradient stagnation, which degrades training stability and performance. 3) Subsequent text-only RL training, following the multimodal RL phase, further enhances multimodal reasoning. This staged training approach effectively balances perceptual grounding and cognitive reasoning development. By incorporating the above insights and addressing multimodal RL issues, we introduce ReVisual-R1, achieving a new state-of-the-art among open-source 7B MLLMs on challenging benchmarks including MathVerse, MathVision, WeMath, LogicVista, DynaMath, and challenging AIME2024 and AIME2025.

Problem

• Optimizing cold start initialization for MLLM reasoning enhancement
• Addressing gradient stagnation in standard GRPO for multimodal RL
• Improving multimodal reasoning via staged text and RL training

Innovation

• Optimized cold start with text data
• Staged reinforcement learning approach
• Addresses gradient stagnation in GRPO