Vision-Language Model Based Multi-Expert Fusion for CT Image Classification

2026-03-16

AI Summary

This study addresses the challenge of insufficient classification robustness in multi-institutional COVID-19 CT imaging, stemming from data distribution shifts, class imbalance, and unknown test-set origins. To tackle this, the authors propose a three-stage source-aware multi-expert fusion framework: first, a lung-aware 3D CNN performs voxel-level classification; second, a MedSigLIP vision-language model integrated with a Transformer captures slice-level representations and cross-slice contextual information; third, a source classifier predicts the origin of test samples to dynamically weight expert outputs. This work pioneers the integration of vision-language models with a source-aware multi-expert mechanism, leveraging a hierarchical voting strategy to significantly enhance generalization across heterogeneous multi-source data. Evaluated on a validation set encompassing four data sources, the model achieves a Macro-F1 score of 0.9711, an AUC of 0.9864, and a source classification accuracy of 0.9107, demonstrating markedly improved robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Security and Privacy: Data transparency and provenance

Abstract

Robust detection of COVID-19 from chest CT remains challenging in multi-institutional settings due to substantial source shift, source imbalance, and hidden test-source identities. In this work, we propose a three-stage source-aware multi-expert framework for multi-source COVID-19 CT classification. First, we build a lung-aware 3D expert by combining original CT volumes and lung-extracted CT volumes for volumetric classification. Second, we develop two MedSigLIP-based experts: a slice-wise representation and probability learning module, and a Transformer-based inter-slice context modeling module for capturing cross-slice dependency. Third, we train a source classifier to predict the latent source identity of each test scan. By leveraging the predicted source information, we perform model fusion and voting based on different experts. On the validation set covering all four sources, the Stage 1 model achieves the best macro-F1 of 0.9711, ACC of 0.9712, and AUC of 0.9791. Stage~2a and Stage~2b achieve the best AUC scores of 0.9864 and 0.9854, respectively. Stage~3 source classifier reaches 0.9107 ACC and 0.9114 F1. These results demonstrate that source-aware expert modeling and hierarchical voting provide an effective solution for robust COVID-19 CT classification under heterogeneous multi-source conditions.

Problem

• source shift
• source imbalance
• hidden test-source identities
• multi-institutional CT classification
• robust COVID-19 detection

Innovation

• multi-expert fusion
• source-aware learning
• MedSigLIP
• 3D lung-aware modeling
• inter-slice context modeling