X2CT-CLIP: Enable Multi-Abnormality Detection in Computed Tomography from Chest Radiography via Tri-Modal Contrastive Learning

2025-03-04

AI Summary

To address the limited clinical adoption of CT-based multi-abnormality detection due to high radiation exposure and cost, this paper proposes a cross-modal, multi-disease joint recognition framework leveraging chest X-ray (CXR) images. We introduce the first tri-modal contrastive alignment framework integrating CT volumes, CXR images, and radiology reports. Adopting a CLIP-style dual-encoder architecture, we incorporate report-supervised latent-space alignment to enable effective knowledge transfer from 3D CT to 2D CXR. Our method overcomes both single-disease constraints and modality gaps. It achieves state-of-the-art performance on three multi-label CT benchmarks. Moreover, it attains superior results in cross-modal retrieval, few-shot adaptation, and external validation—demonstrating robust generalization. This work establishes a new paradigm for low-radiation, resource-efficient disease screening in settings with constrained access to advanced imaging.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Computed tomography (CT) is a key imaging modality for diagnosis, yet its clinical utility is marred by high radiation exposure and long turnaround times, restricting its use for larger-scale screening. Although chest radiography (CXR) is more accessible and safer, existing CXR foundation models focus primarily on detecting diseases that are readily visible on the CXR. Recently, works have explored training disease classification models on simulated CXRs, but they remain limited to recognizing a single disease type from CT. CT foundation models have also emerged with significantly improved detection of pathologies in CT. However, the generalized application of CT-derived labels on CXR has remained illusive. In this study, we propose X2CT-CLIP, a tri-modal knowledge transfer learning framework that bridges the modality gap between CT and CXR while reducing the computational burden of model training. Our approach is the first work to enable multi-abnormality classification in CT, using CXR, by transferring knowledge from 3D CT volumes and associated radiology reports to a CXR encoder via a carefully designed tri-modal alignment mechanism in latent space. Extensive evaluations on three multi-label CT datasets demonstrate that our method outperforms state-of-the-art baselines in cross-modal retrieval, few-shot adaptation, and external validation. These results highlight the potential of CXR, enriched with knowledge derived from CT, as a viable efficient alternative for disease detection in resource-limited settings.

Problem

• Bridges CT and CXR modality gap for disease detection
• Enables multi-abnormality classification in CT using CXR
• Reduces computational burden in cross-modal model training

Innovation

• Tri-modal knowledge transfer learning framework
• Bridges CT and CXR modality gap
• Enables multi-abnormality classification via CXR