CoM$^3$eT: A foundation model for medical image analysis through federated, multidimensional context integration

2026-08-17

AI Summary

This study addresses the fragmentation of specialties and limited modality support in existing medical foundation models by proposing a unified visual foundation model integrating pathology and radiology. Leveraging a multi-dimensional context attention mechanism, the framework unifies sparse and dense prediction tasks across 2D and high-dimensional inputs. Combined with multi-task joint training and parameter-efficient fine-tuning (PEFT), it enables federated learning on consumer-grade hardware. This work represents the first cross-specialty, multi-dimensional unified modeling approach for medical vision, achieving state-of-the-art performance across 12 benchmark datasets. Notably, fine-tuning less than 2.5% of parameters yields results comparable to full fine-tuning, while federated learning performance closely approximates centralized training, significantly enhancing adaptation efficiency in low-resource settings.

Abstract

Medical foundation models improve generalization when training AI models with limited labeled data, but remain confined to a single specialty, such as pathology or radiology, and to either sparse or dense outputs, such as classification or segmentation. Here, we present CoM$^3$eT (Co-representation Multidimensional Multitask Medical Transformer), a medical vision foundation model that unifies pathology and radiology, sparse and dense predictions, and two- and higher-dimensional inputs by modeling multidimensional context with attention. CoM$^3$eT outperformed other medical foundation models in an open competition spanning five tomographic, four whole-specimen, and three two-dimensional datasets, covering sparse and dense prediction tasks as well as report generation. When adapted across diverse clinical applications, training fewer than 2.5% of parameters achieved performance comparable to full fine-tuning, enabling research without access to high-performance GPU clusters. Applied to federated learning across hospitals, this approach achieved performance comparable to pooled-data training over internet connections and with consumer-grade hardware.

Problem

• Medical Foundation Model
• Multidimensional Context Integration
• Federated Learning
• Cross-specialty Generalization
• Sparse and Dense Predictions

Innovation

• Medical Foundation Model
• Multidimensional Context Integration
• Parameter-Efficient Fine-Tuning
• Federated Learning
• Cross-Domain Unification