GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking

2025-06-01

AI Summary

Current multimodal large language models (MLLMs) underperform on vision-dominant multimodal reasoning tasks, primarily due to overreliance on logical and knowledge-driven “slow thinking” while neglecting dynamic integration and polysemous interpretation of visual cues. To address this, we propose Cue-Rethinking—a novel paradigm enabling iterative vision-language grounding via a two-stage training strategy: modality-guided cold-start initialization followed by reward-driven reinforcement learning. We introduce GThinker-11K, the first high-quality, human-annotated dataset tailored for general multimodal reasoning, constructed through synergistic human labeling and RL-generated sample curation. Evaluated on the M³CoT benchmark, our method achieves 81.5%, surpassing O4-mini; it yields an average 2.1% improvement across general multimodal reasoning tasks and matches state-of-the-art performance on mathematical reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Despite notable advancements in multimodal reasoning, leading Multimodal Large Language Models (MLLMs) still underperform on vision-centric multimodal reasoning tasks in general scenarios. This shortfall stems from their predominant reliance on logic- and knowledge-based slow thinking strategies, while effective for domains like math and science, fail to integrate visual information effectively during reasoning. Consequently, these models often fail to adequately ground visual cues, resulting in suboptimal performance in tasks that require multiple plausible visual interpretations and inferences. To address this, we present GThinker (General Thinker), a novel reasoning MLLM excelling in multimodal reasoning across general scenarios, mathematics, and science. GThinker introduces Cue-Rethinking, a flexible reasoning pattern that grounds inferences in visual cues and iteratively reinterprets these cues to resolve inconsistencies. Building on this pattern, we further propose a two-stage training pipeline, including pattern-guided cold start and incentive reinforcement learning, designed to enable multimodal reasoning capabilities across domains. Furthermore, to support the training, we construct GThinker-11K, comprising 7K high-quality, iteratively-annotated reasoning paths and 4K curated reinforcement learning samples, filling the data gap toward general multimodal reasoning. Extensive experiments demonstrate that GThinker achieves 81.5% on the challenging comprehensive multimodal reasoning benchmark M$^3$CoT, surpassing the latest O4-mini model. It also shows an average improvement of 2.1% on general scenario multimodal reasoning benchmarks, while maintaining on-par performance in mathematical reasoning compared to counterpart advanced reasoning models. The code, model, and data will be released soon at https://github.com/jefferyZhan/GThinker.

Problem

• Improving vision-centric multimodal reasoning in general scenarios
• Addressing inadequate visual cue integration in MLLMs
• Enhancing performance in tasks with multiple visual interpretations

Innovation

• Cue-Rethinking for visual cue integration
• Two-stage training pipeline for reasoning
• GThinker-11K dataset for multimodal training