Evidence-based diagnostic reasoning with multi-agent copilot for human pathology

2025-06-25

AI Summary

Current computational pathology faces three critical bottlenecks: (1) deep learning models lack instruction-following capability and contextual multimodal fusion; (2) multimodal large language models (MLLMs) suffer from scarcity of annotated pathology data, inadequate support for multi-image reasoning, and absence of autonomous diagnostic reasoning; and (3) existing frameworks cannot handle gigapixel whole-slide images (WSIs) with hierarchical, interpretable inference. To address these, we propose PathChat+, introducing the first million-scale pathology instruction-tuning dataset and SlideSeek—a novel multi-agent system enabling hierarchical, autonomous WSI diagnosis. Our method integrates vision-language alignment, instruction tuning, and hierarchical reasoning to support natural-language interaction and generation of interpretable, multimodal diagnostic reports. Evaluated on benchmarks including DDxBench, PathChat+ significantly outperforms state-of-the-art models, achieving differential diagnosis accuracy at clinically viable levels.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Pathology is experiencing rapid digital transformation driven by whole-slide imaging and artificial intelligence (AI). While deep learning-based computational pathology has achieved notable success, traditional models primarily focus on image analysis without integrating natural language instruction or rich, text-based context. Current multimodal large language models (MLLMs) in computational pathology face limitations, including insufficient training data, inadequate support and evaluation for multi-image understanding, and a lack of autonomous, diagnostic reasoning capabilities. To address these limitations, we introduce PathChat+, a new MLLM specifically designed for human pathology, trained on over 1 million diverse, pathology-specific instruction samples and nearly 5.5 million question answer turns. Extensive evaluations across diverse pathology benchmarks demonstrated that PathChat+ substantially outperforms the prior PathChat copilot, as well as both state-of-the-art (SOTA) general-purpose and other pathology-specific models. Furthermore, we present SlideSeek, a reasoning-enabled multi-agent AI system leveraging PathChat+ to autonomously evaluate gigapixel whole-slide images (WSIs) through iterative, hierarchical diagnostic reasoning, reaching high accuracy on DDxBench, a challenging open-ended differential diagnosis benchmark, while also capable of generating visually grounded, humanly-interpretable summary reports.

Problem

• Lack of natural language integration in pathology AI models
• Insufficient multi-image understanding in current MLLMs
• Absence of autonomous diagnostic reasoning in computational pathology

Innovation

• PathChat+ MLLM trained on pathology-specific data
• SlideSeek system for autonomous WSI evaluation
• Iterative hierarchical reasoning for accurate diagnosis