multiGradICON: A Foundation Model for Multimodal Medical Image Registration

2024-08-01

Workshop on Biomedical Image Registration

AI Summary

Existing medical image registration methods are predominantly limited to single-modality scenarios and lack generalizability, rendering multimodal registration particularly challenging. This paper introduces UniReg, the first foundation model for unified single- and multimodal image registration, which overcomes modality-specific barriers to enable anatomical alignment across diverse imaging modalities. Methodologically, UniReg builds upon the GradICON framework to construct a gradient-guided deformable registration backbone; proposes a novel randomized multi-task loss strategy to mitigate inter-modal distribution shifts; and introduces a cross-modal joint training paradigm to substantially enhance generalization. Evaluated on multiple cross-modal benchmarks—including MRI–CT and MRI–PET—UniReg achieves state-of-the-art accuracy while preserving competitive single-modal performance. The code and pretrained models are publicly released.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Modern medical image registration approaches predict deformations using deep networks. These approaches achieve state-of-the-art (SOTA) registration accuracy and are generally fast. However, deep learning (DL) approaches are, in contrast to conventional non-deep-learning-based approaches, anatomy-specific. Recently, a universal deep registration approach, uniGradICON, has been proposed. However, uniGradICON focuses on monomodal image registration. In this work, we therefore develop multiGradICON as a first step towards universal *multimodal* medical image registration. Specifically, we show that 1) we can train a DL registration model that is suitable for monomodal *and* multimodal registration; 2) loss function randomization can increase multimodal registration accuracy; and 3) training a model with multimodal data helps multimodal generalization. Our code and the multiGradICON model are available at https://github.com/uncbiag/uniGradICON.

Problem

• Develops universal multimodal medical image registration
• Enhances accuracy with loss function randomization
• Improves generalization through multimodal data training

Innovation

• Multimodal medical image registration
• Loss function randomization technique
• Multimodal data training generalization