Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models

2025-05-29

AI Summary

Multi-modal small language models (MSLMs) face three key challenges in multi-modal reasoning: scarcity of high-quality training data, degradation of textual logical reasoning due to visual integration, and error propagation in reinforcement learning (RL) caused by complex erroneous reasoning chains. To address these, we propose a three-stage curriculum-based RL framework: first activating textual logical reasoning, then transferring this capability to cross-modal scenarios, and finally enhancing caption-free multi-modal mathematical reasoning. Our contributions are threefold: (1) the first capability-structured three-phase curriculum training paradigm for MSLMs; (2) the first robust transfer of textual reasoning competence to pure visual multi-modal reasoning; and (3) effective mitigation of language bias—enhancing both correctness and interpretability—via rule-guided RL, caption-augmented and caption-free data synergy, and multi-stage capability decoupling and alignment. Infi-MMR-3B achieves new state-of-the-art performance on MathVerse, MathVision, OlympiadBench, and MathVista.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advancements in large language models (LLMs) have demonstrated substantial progress in reasoning capabilities, such as DeepSeek-R1, which leverages rule-based reinforcement learning to enhance logical reasoning significantly. However, extending these achievements to multimodal large language models (MLLMs) presents critical challenges, which are frequently more pronounced for Multimodal Small Language Models (MSLMs) given their typically weaker foundational reasoning abilities: (1) the scarcity of high-quality multimodal reasoning datasets, (2) the degradation of reasoning capabilities due to the integration of visual processing, and (3) the risk that direct application of reinforcement learning may produce complex yet incorrect reasoning processes. To address these challenges, we design a novel framework Infi-MMR to systematically unlock the reasoning potential of MSLMs through a curriculum of three carefully structured phases and propose our multimodal reasoning model Infi-MMR-3B. The first phase, Foundational Reasoning Activation, leverages high-quality textual reasoning datasets to activate and strengthen the model's logical reasoning capabilities. The second phase, Cross-Modal Reasoning Adaptation, utilizes caption-augmented multimodal data to facilitate the progressive transfer of reasoning skills to multimodal contexts. The third phase, Multimodal Reasoning Enhancement, employs curated, caption-free multimodal data to mitigate linguistic biases and promote robust cross-modal reasoning. Infi-MMR-3B achieves both state-of-the-art multimodal math reasoning ability (43.68% on MathVerse testmini, 27.04% on MathVision test, and 21.33% on OlympiadBench) and general reasoning ability (67.2% on MathVista testmini).

Problem

• Addressing scarcity of high-quality multimodal reasoning datasets
• Preventing reasoning degradation from visual processing integration
• Avoiding incorrect complex reasoning from direct reinforcement learning

Innovation

• Curriculum-based phased reinforcement learning approach
• High-quality textual reasoning datasets activation
• Caption-free multimodal data for bias mitigation