Generalist versus Specialist Vision Foundation Models for Ocular Disease and Oculomics

2025-09-03

AI Summary

This study investigates the transfer performance disparity between general-purpose vision foundation models (DINOv2/DINOv3) and retinal domain-specific models (RETFound-MAE/RETFound-DINOv2) on ophthalmic disease detection and ocular omics tasks. Systematically comparing fine-tuning and linear probing as transfer strategies, we find that although general models improve with scale, RETFound-DINOv2 consistently outperforms them across most tasks—especially under low-data regimes—demonstrating superior data efficiency, adaptation efficiency, and cross-task generalization. Our key contribution is the first quantitative demonstration of persistent gains from retinal-domain pretraining, establishing the irreplaceability of domain-specialized representations in medical imaging downstream tasks and providing empirical support for the design paradigm of clinical foundation models.

Technology Category

• Computer Vision: Other Foundations of Computer Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Medical foundation models, pre-trained with large-scale clinical data, demonstrate strong performance in diverse clinically relevant applications. RETFound, trained on nearly one million retinal images, exemplifies this approach in applications with retinal images. However, the emergence of increasingly powerful and multifold larger generalist foundation models such as DINOv2 and DINOv3 raises the question of whether domain-specific pre-training remains essential, and if so, what gap persists. To investigate this, we systematically evaluated the adaptability of DINOv2 and DINOv3 in retinal image applications, compared to two specialist RETFound models, RETFound-MAE and RETFound-DINOv2. We assessed performance on ocular disease detection and systemic disease prediction using two adaptation strategies: fine-tuning and linear probing. Data efficiency and adaptation efficiency were further analysed to characterise trade-offs between predictive performance and computational cost. Our results show that although scaling generalist models yields strong adaptability across diverse tasks, RETFound-DINOv2 consistently outperforms these generalist foundation models in ocular-disease detection and oculomics tasks, demonstrating stronger generalisability and data efficiency. These findings suggest that specialist retinal foundation models remain the most effective choice for clinical applications, while the narrowing gap with generalist foundation models suggests that continued data and model scaling can deliver domain-relevant gains and position them as strong foundations for future medical foundation models.

Problem

• Evaluating adaptability of generalist vs specialist vision models
• Assessing performance in ocular disease detection tasks
• Comparing data efficiency and computational cost trade-offs

Innovation

• Specialist RETFound-DINOv2 outperforms generalist models
• Systematic evaluation of adaptation strategies fine-tuning
• Domain-specific pre-training maintains clinical effectiveness superiority