Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Clinical Pathology Analysis

2024-12-12

arXiv.org

AI Summary

To address the inefficiency and reduced diagnostic accuracy arising from redundant multi-scale feature extraction in whole-slide images (WSIs) and the resolution limitations of large vision-language models (LVLMs), this work proposes a dual-strategy framework: (1) hybrid task-guided feature enhancement for precise focusing on lesion-relevant multi-scale features, and (2) prompt-driven detail completion for coarse-to-fine collaborative modeling. We introduce OmniPath—a specialized LVLM trained on 490,000 pathology samples—incorporating multi-task supervision, hierarchical feature alignment, lightweight prompt modulation, and high-resolution WSI adaptation. Evaluated on cancer detection, histopathological grading, and vascular/peri-neural invasion identification, OmniPath achieves an 8.2% improvement in diagnostic accuracy while enabling real-time interactive inference. The model has been deployed in clinical settings for AI-assisted diagnosis.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Pathological diagnosis is vital for determining disease characteristics, guiding treatment, and assessing prognosis, relying heavily on detailed, multi-scale analysis of high-resolution whole slide images (WSI). However, traditional pure vision models face challenges of redundant feature extraction, whereas existing large vision-language models (LVLMs) are limited by input resolution constraints, hindering their efficiency and accuracy. To overcome these issues, we propose two innovative strategies: the mixed task-guided feature enhancement, which directs feature extraction toward lesion-related details across scales, and the prompt-guided detail feature completion, which integrates coarse- and fine-grained features from WSI based on specific prompts without compromising inference speed. Leveraging a comprehensive dataset of 490,000 samples from diverse pathology tasks-including cancer detection, grading, vascular and neural invasion identification, and so on-we trained the pathology-specialized LVLM, OmniPath. Extensive experiments demonstrate that this model significantly outperforms existing methods in diagnostic accuracy and efficiency, offering an interactive, clinically aligned approach for auxiliary diagnosis in a wide range of pathology applications.

Problem

• Overcoming redundant feature extraction in traditional vision models
• Addressing input resolution limits in large vision-language models
• Enhancing clinical pathology analysis accuracy and efficiency

Innovation

• Mixed task-guided feature enhancement for lesion details
• Prompt-guided detail feature completion from WSI
• OmniPath LVLM for clinical pathology accuracy