Building 3D In-Context Learning Universal Model in Neuroimaging

2025-03-04

AI Summary

Existing 2D image-based in-context learning (ICL) models struggle to capture the intrinsic 3D anatomical structure of neuroimaging data, resulting in weak global perception and poor generalization. To address this, we propose Neuroverse3D—the first general-purpose ICL model supporting multi-task 3D neuroimaging inference without fine-tuning, capable of performing 14 diverse tasks including segmentation, denoising, and inpainting. Methodologically, we introduce an adaptive parallel-sequential context modeling mechanism and a U-shaped cross-scale feature fusion architecture, integrated with an anatomy-aware multi-task optimization loss. A 3D vision transformer enables efficient volumetric representation learning. Extensive experiments across 19 datasets comprising 43,674 3D scans demonstrate that Neuroverse3D significantly outperforms existing ICL approaches and approaches the performance of task-specific models. The code and pre-trained weights are publicly available.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In-context learning (ICL), a type of universal model, demonstrates exceptional generalization across a wide range of tasks without retraining by leveraging task-specific guidance from context, making it particularly effective for the complex demands of neuroimaging. However, existing ICL models, which take 2D images as input, struggle to fully leverage the 3D anatomical structures in neuroimages, leading to a lack of global awareness and suboptimal performance. In this regard, we introduce Neuroverse3D, an ICL model capable of performing multiple neuroimaging tasks (e.g., segmentation, denoising, inpainting) in 3D. Neuroverse3D overcomes the large memory consumption due to 3D inputs through adaptive parallel-sequential context processing and a U-shape fusion strategy, allowing it to handle an unlimited number of context images. Additionally, we propose an optimized loss to balance multi-task training and enhance the focus on anatomical structures. Our study incorporates 43,674 3D scans from 19 neuroimaging datasets and evaluates Neuroverse3D on 14 diverse tasks using held-out test sets. The results demonstrate that Neuroverse3D significantly outperforms existing ICL models and closely matches the performance of task-specific models. The code and model weights are publicly released at: https://github.com/jiesihu/Neu3D.

Problem

• Develops a 3D in-context learning model for neuroimaging tasks.
• Addresses limitations of 2D models in leveraging 3D anatomical structures.
• Optimizes memory usage and multi-task training for enhanced performance.

Innovation

• 3D In-Context Learning for neuroimaging tasks
• Adaptive parallel-sequential context processing
• U-shape fusion strategy for memory efficiency