GRIT: Teaching MLLMs to Think with Images

2025-05-21

AI Summary

Existing open-source visual reasoning models rely solely on text generation, leading to a disconnection between linguistic and visual information and undermining the visual grounding of reasoning chains. Method: We propose “embodied multimodal reasoning with interleaved vision-language representations,” which dynamically integrates natural language tokens and image region coordinates into reasoning chains to achieve fine-grained cross-modal alignment. Our approach introduces the first image-text interleaved reasoning chain generation mechanism; designs GRPO-GR—a reinforcement learning algorithm requiring no human-annotated reasoning chains or bounding boxes; and enables efficient training with only 20 annotated samples. Key technical components include coordinate-aware modeling, visual grounding, and multimodal prompt optimization. Contribution/Results: Our method significantly improves both visual interpretability and logical coherence of reasoning chains, achieving state-of-the-art performance across multiple visual question answering and reasoning benchmarks—marking the first framework that unifies strong reasoning capability with precise visual localization.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent studies have demonstrated the efficacy of using Reinforcement Learning (RL) in building reasoning models that articulate chains of thoughts prior to producing final answers. However, despite ongoing advances that aim at enabling reasoning for vision-language tasks, existing open-source visual reasoning models typically generate reasoning content with pure natural language, lacking explicit integration of visual information. This limits their ability to produce clearly articulated and visually grounded reasoning chains. To this end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method for training MLLMs to think with images. GRIT introduces a grounded reasoning paradigm, in which models generate reasoning chains that interleave natural language and explicit bounding box coordinates. These coordinates point to regions of the input image that the model consults during its reasoning process. Additionally, GRIT is equipped with a reinforcement learning approach, GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused on the final answer accuracy and format of the grounded reasoning output, which eliminates the need for data with reasoning chain annotations or explicit bounding box labels. As a result, GRIT achieves exceptional data efficiency, requiring as few as 20 image-question-answer triplets from existing datasets. Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to produce coherent and visually grounded reasoning chains, showing a successful unification of reasoning and grounding abilities.

Problem

• Enabling visual reasoning with integrated image-text chains
• Eliminating need for annotated reasoning chain data
• Training MLLMs efficiently with minimal data input

Innovation

• Interleaves natural language with bounding boxes
• Uses GRPO-GR reinforcement learning for efficiency
• Requires minimal annotated data for training