Vision Foundation Models for Computed Tomography

2025-01-15

AI Summary

The absence of dedicated 3D foundation models for CT imaging hinders generalizable and clinically deployable AI in radiology. Method: We propose CT-FM—the first open-source 3D vision foundation model specifically designed for CT scans. Trained via self-supervised contrastive learning on 148,000 unlabeled CT volumes, CT-FM supports diverse downstream tasks including whole-body/tumor segmentation, cranial CT triage, medical image retrieval, and semantic understanding. Contribution/Results: CT-FM introduces novel cross-scan anatomical clustering and concept alignment, achieving robust test–retest reliability and interpretable embedding spaces. It outperforms all state-of-the-art methods across four major radiological task categories. To foster clinical translation, we fully open-source the model weights, training code, and preprocessing pipelines. CT-FM advances adaptable, reliable, and interpretable AI for real-world radiology practice.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Foundation models (FMs) have shown transformative potential in radiology by performing diverse, complex tasks across imaging modalities. Here, we developed CT-FM, a large-scale 3D image-based pre-trained model designed explicitly for various radiological tasks. CT-FM was pre-trained using 148,000 computed tomography (CT) scans from the Imaging Data Commons through label-agnostic contrastive learning. We evaluated CT-FM across four categories of tasks, namely, whole-body and tumor segmentation, head CT triage, medical image retrieval, and semantic understanding, showing superior performance against state-of-the-art models. Beyond quantitative success, CT-FM demonstrated the ability to cluster regions anatomically and identify similar anatomical and structural concepts across scans. Furthermore, it remained robust across test-retest settings and indicated reasonable salient regions attached to its embeddings. This study demonstrates the value of large-scale medical imaging foundation models and by open-sourcing the model weights, code, and data, aims to support more adaptable, reliable, and interpretable AI solutions in radiology.

Problem

• 3D Vision Model
• CT Scan Images
• Radiology Assistance

Innovation

• CT-FM Model
• 3D Image Processing
• Medical Imaging Analysis