MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis

2025-06-10

AI Summary

Current ophthalmic AI models rely heavily on large-scale annotated datasets, exhibit poor cross-center generalization, and lack rigorously validated multimodal foundation models—especially for segmentation tasks. To address these limitations, we propose RetinaFM, the first general-purpose foundation model for retinal OCT and SLO imaging. RetinaFM employs multimodal self-supervised pretraining, cross-modal feature alignment, a unified decoder, and a mask-driven segmentation head to enable few-shot adaptation and robust generalization. Concurrently, we introduce RetinaBench, a standardized benchmark covering both classification and segmentation tasks. Extensive experiments demonstrate that RetinaFM consistently outperforms general-purpose and unimodal ophthalmic models in lesion identification and pixel-level segmentation on OCT/SLO data, while achieving significant gains in cross-center generalization. The code, pretrained models, and benchmark are publicly released.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Artificial intelligence (AI) has become a fundamental tool for assisting clinicians in analyzing ophthalmic images, such as optical coherence tomography (OCT). However, developing AI models often requires extensive annotation, and existing models tend to underperform on independent, unseen data. Foundation models (FMs), large AI models trained on vast unlabeled datasets, have shown promise in overcoming these challenges. Nonetheless, available FMs for ophthalmology lack extensive validation, especially for segmentation tasks, and focus on a single imaging modality. In this context, we propose MIRAGE, a novel multimodal FM for the analysis of OCT and scanning laser ophthalmoscopy (SLO) images. Additionally, we propose a new evaluation benchmark with OCT/SLO classification and segmentation tasks. The comparison with general and specialized FMs and segmentation methods shows the superiority of MIRAGE in both types of tasks, highlighting its suitability as a basis for the development of robust AI systems for retinal OCT image analysis. Both MIRAGE and the evaluation benchmark are publicly available: https://github.com/j-morano/MIRAGE.

Problem

• Develops multimodal foundation model for retinal OCT/SLO analysis
• Addresses lack of validated FMs for ophthalmology segmentation tasks
• Proposes benchmark to evaluate OCT/SLO classification and segmentation

Innovation

• Multimodal foundation model for OCT and SLO images
• New benchmark for OCT/SLO classification and segmentation
• Publicly available model and evaluation benchmark