Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergy

2026-07-10

AI Summary

Current medical image understanding models struggle to simultaneously achieve strong generalization and fine-grained diagnostic accuracy: general-purpose vision-language models lack precise perception of anatomical structures and lesions, while specialized models suffer from limited generalizability. To address this, this work proposes the Super-Generalist (SuG) framework, which innovatively integrates general vision-language learning with spatial priors derived from multiple expert segmentation models—encompassing anatomical, category-specific, and category-agnostic lesion information. SuG employs a lesion mask–guided attention calibration mechanism to enhance lesion localization and semantic alignment. Evaluated on CT-RATE, Merlin, MedVL-CT69K, and several internal tumor datasets, SuG surpasses specialized models and demonstrates state-of-the-art generalization and specialist-level diagnostic performance across both supervised and unsupervised lesion categories.

Abstract

Medical images require comprehensive and accurate interpretation to support the diagnosis of diverse clincial conditions. Recent vision-language generalist models offer broad task coverage and promising zero-shot capabilities, yet often lack fine-grained anatomical and lesion awareness for reliable diagnosis and spatial interpretability. In contrast, supervised specialist models achieve strong performance on specific tasks but typically lack generalization across diseases and anatomies. In this work, we present SuG, a Super-Generalist framework that unifies generalist vision-language learning with specialist objectives, enabling both broad generalization and specialist-level diagnostic capability. We perform specialist-enhanced vision-language alignment in SuG by incorporating spatial priors from multiple segmentation experts, including anatomy, class-specific lesion and class-agnostic lesion segmentors that captures lesions beyond anatomies annotated during training. To improve lesion grounding capability, we leverage lesion masks as spatial priors to calibrate text-conditioned visual attention, encouraging disease-related semantics to focus on clinically relevant regions. We evaluate SuG on extensive chest and abdominal CT benchmarks, including CT-RATE, Merlin, MedVL-CT69K, and several in-house tumor datasets. SuG achieves state-of-the-art performance across a wide range of disease diagnosis tasks and surpasses specialist models on several critical tumor diagnosis benchmarks. Furthermore, SuG demonstrates strong lesion grounding capability, including robust generalization to lesion types lacking class-specific supervision.

Problem

• medical image understanding
• generalist-specialist synergy
• lesion grounding
• zero-shot diagnosis
• spatial interpretability

Innovation

• vision-language alignment
• spatial priors
• lesion grounding
• generalist-specialist synergy
• medical image understanding