A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model

2025-07-23

AI Summary

Existing pathological multimodal large language models (MLLMs) exhibit weak reasoning capabilities, rely on costly chain-of-thought annotations, and support only region-of-interest (ROI)-level visual question answering (VQA), failing to meet diverse clinical diagnostic needs—including ROI classification, detection, segmentation, whole-slide image (WSI) classification, and VQA. Method: We propose SmartPath-R1—the first pathological MLLM supporting both ROI-level and WSI-level multi-task learning (classification, detection, segmentation, VQA). It introduces a scale-aware supervised fine-tuning framework and a task-aware reinforcement fine-tuning framework, coupled with a multi-expert dynamic collaboration mechanism—eliminating the need for chain-of-thought annotations. Contribution/Results: Trained on a large-scale pathology dataset comprising 2.3 million ROI samples and 188,000 WSIs, SmartPath-R1 achieves state-of-the-art performance across 72 benchmark tasks, demonstrating significant improvements over prior methods and strong potential for clinical deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal large language models (MLLMs) have emerged as powerful tools for computational pathology, offering unprecedented opportunities to integrate pathological images with language context for comprehensive diagnostic analysis. These models hold particular promise for automating complex tasks that traditionally require expert interpretation of pathologists. However, current MLLM approaches in pathology demonstrate significantly constrained reasoning capabilities, primarily due to their reliance on expensive chain-of-thought annotations. Additionally, existing methods remain limited to simplex application of visual question answering (VQA) at region-of-interest (ROI) level, failing to address the full spectrum of diagnostic needs such as ROI classification, detection, segmentation, whole-slide-image (WSI) classification and VQA in clinical practice. In this study, we present SmartPath-R1, a versatile MLLM capable of simultaneously addressing both ROI-level and WSI-level tasks while demonstrating robust pathological reasoning capability. Our framework combines scale-dependent supervised fine-tuning and task-aware reinforcement fine-tuning, which circumvents the requirement for chain-of-thought supervision by leveraging the intrinsic knowledge within MLLM. Furthermore, SmartPath-R1 integrates multiscale and multitask analysis through a mixture-of-experts mechanism, enabling dynamic processing for diverse tasks. We curate a large-scale dataset comprising 2.3M ROI samples and 188K WSI samples for training and evaluation. Extensive experiments across 72 tasks validate the effectiveness and superiority of the proposed approach. This work represents a significant step toward developing versatile, reasoning-enhanced AI systems for precision pathology.

Problem

• Enhancing reasoning in pathology MLLMs without costly annotations
• Expanding MLLM capabilities beyond ROI-level VQA to WSI tasks
• Integrating multiscale analysis for diverse diagnostic needs in pathology

Innovation

• Combines scale-dependent and task-aware fine-tuning
• Uses mixture-of-experts for multiscale multitask analysis
• Leverages intrinsic MLLM knowledge for reasoning