MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision

2025-08-11

AI Summary

Current medical multimodal large language models (MLLMs) rely on explicit spatial annotations for region-of-interest (ROI) localization, limiting their applicability to clinically prevalent implicit queries. To address this, we propose the “Unified Medical Reasoning and Localization” task and introduce the first reasoning-segmentation decoupled framework tailored for implicit queries. Our method pioneers the integration of reinforcement learning into medical vision-language modeling, featuring a dual reward mechanism that jointly optimizes output format fidelity and localization accuracy. Leveraging the 14K high-quality reasoning trajectory dataset U-MRG-14K, our approach achieves end-to-end ROI localization without pixel-level supervision. It employs a frozen segmentation expert as the localizer and an MLLM as the reasoning engine, significantly improving generalization to unseen clinical queries. Evaluated across multiple metrics, our method establishes new state-of-the-art performance, enhancing both accuracy and interpretability in medical image understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Accurately grounding regions of interest (ROIs) is critical for diagnosis and treatment planning in medical imaging. While multimodal large language models (MLLMs) combine visual perception with natural language, current medical-grounding pipelines still rely on supervised fine-tuning with explicit spatial hints, making them ill-equipped to handle the implicit queries common in clinical practice. This work makes three core contributions. We first define Unified Medical Reasoning Grounding (UMRG), a novel vision-language task that demands clinical reasoning and pixel-level grounding. Second, we release U-MRG-14K, a dataset of 14K samples featuring pixel-level masks alongside implicit clinical queries and reasoning traces, spanning 10 modalities, 15 super-categories, and 108 specific categories. Finally, we introduce MedReasoner, a modular framework that distinctly separates reasoning from segmentation: an MLLM reasoner is optimized with reinforcement learning, while a frozen segmentation expert converts spatial prompts into masks, with alignment achieved through format and accuracy rewards. MedReasoner achieves state-of-the-art performance on U-MRG-14K and demonstrates strong generalization to unseen clinical queries, underscoring the significant promise of reinforcement learning for interpretable medical grounding.

Problem

• Accurately grounding ROIs in medical imaging for diagnosis
• Handling implicit clinical queries without spatial hints
• Unifying clinical reasoning with pixel-level segmentation

Innovation

• Reinforcement learning optimizes MLLM reasoning
• Modular framework separates reasoning and segmentation
• Frozen segmentation expert converts prompts to masks