Triad: Vision Foundation Model for 3D Magnetic Resonance Imaging

2025-02-19

AI Summary

To address the poor generalizability of CT-pretrained vision foundation models on 3D MRI tasks, this work introduces the first vision foundation model specifically designed for 3D MRI. Methodologically: (1) we construct Triad-131K, the largest publicly available 3D MRI pretraining dataset to date, comprising 131,170 volumetric scans; (2) we propose an autoencoder-based architecture incorporating natural language–guided, organ-agnostic semantic constraints to learn modality-consistent, general-purpose representations; and (3) the model supports seamless transfer to mainstream downstream architectures—including nnUNet, Swin-B, and SwinUNETR. Evaluated across 17 segmentation, 5 classification, and 2 registration benchmarks, our MRI-specific foundation model achieves average performance gains of +6.88%, +3.97%, and +4.00%, respectively, demonstrating that MRI-tailored pretraining significantly enhances cross-task generalization.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Vision foundation models (VFMs) are pre-trained on extensive image datasets to learn general representations for diverse types of data. These models can subsequently be fine-tuned for specific downstream tasks, significantly boosting performance across a broad range of applications. However, existing vision foundation models that claim to be applicable to various radiology tasks are mostly pre-trained on 3D computed tomography (CT), which benefits from the availability of extensive 3D CT databases. Significant differences between CT and magnetic resonance imaging (MRI) in imaging principles, signal characteristics, and data distribution may hinder their practical performance and versatility in MRI-specific applications. Here, we propose Triad, a vision foundation model for 3D MRI. Triad adopts a widely used autoencoder architecture to learn robust representations from 131,170 3D MRI volumes and uses organ-independent imaging descriptions to constrain the semantic distribution of the visual modality. The above pre-training dataset is called Triad-131K, which is currently the largest 3D MRI pre-training dataset. We evaluate Triad across three tasks, namely, organ/tumor segmentation, organ/cancer classification, and medical image registration, in two data modalities (within-domain and out-of-domain) settings using 25 downstream datasets. By initializing models with Triad's pre-trained weights, nnUNet-Triad improves segmentation performance by 6.88% compared to nnUNet-Scratch across 17 datasets. Swin-B-Triad achieves a 3.97% improvement over Swin-B-Scratch in classification tasks across five datasets. SwinUNETR-Triad improves by 4.00% compared to SwinUNETR-Scratch in registration tasks across two datasets. Our study demonstrates that pre-training can maximize performance when the data modalities and organs of upstream and downstream tasks are consistent.

Problem

• Develops a vision foundation model for 3D MRI
• Addresses limitations of CT-based models in MRI applications
• Improves performance in medical image tasks with pre-training

Innovation

• Autoencoder architecture for MRI
• Organ-independent imaging descriptions
• Largest 3D MRI pre-training dataset