Specialized curricula for training vision-language models in retinal image analysis

2024-07-11

AI Summary

Current vision-language models (VLMs) exhibit limited clinical utility in retinal image analysis, particularly for age-related macular degeneration (AMD) staging and referral decision-making—significantly underperforming relative to ophthalmologists. To address this gap, we propose RetinaVLM: the first AMD-specific clinical capability taxonomy, introducing a novel ophthalmology-driven VLM capability disentanglement framework and a progressive curriculum-based fine-tuning paradigm. Leveraging multi-source, image–report aligned datasets, RetinaVLM integrates curriculum-guided instruction tuning with domain-enhanced alignment techniques, adaptable to both ChatGPT-4o and medical VLM architectures. Experiments demonstrate substantial improvements over baselines: F1-scores of 0.63 for AMD staging and 0.67 for referral recommendation. In double-blind clinical evaluation, RetinaVLM achieves 64.3% report accuracy—markedly surpassing ChatGPT-4o (14.3%)—and attains performance comparable to junior ophthalmologists for the first time.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Clinicians spend a significant amount of time reviewing medical images and transcribing their findings regarding patient diagnosis, referral and treatment in text form. Vision-language models (VLMs), which automatically interpret images and summarize their findings as text, have enormous potential to alleviate clinical workloads and increase patient access to high-quality medical care. While foundational models have stirred considerable interest in the medical community, it is unclear whether their general capabilities translate to real-world clinical utility. In this work, we demonstrate that OpenAI's ChatGPT-4o model, in addition to two foundation VLMs designed for medical use, markedly underperform compared to practicing ophthalmologists on specialist tasks crucial to the care of patients with age-related macular degeneration (AMD). To address this, we initially identified the essential capabilities required for image-based clinical decision-making, and then developed a curriculum to selectively train VLMs in these skills. The resulting model, RetinaVLM, can be instructed to write reports that significantly outperform those written by leading foundation medical VLMs and ChatGPT-4o in disease staging (F1 score of 0.63 vs. 0.33) and patient referral (0.67 vs. 0.50), and approaches the diagnostic performance of junior ophthalmologists (who achieve 0.77 and 0.78 on the respective tasks). Furthermore, in a single-blind reader study two senior ophthalmologists with up to 32 years of experience found RetinaVLM's reports were found to be substantially more accurate than those by ChatGPT-4o (64.3% vs. 14.3%). These results reinforce that our curriculum-based approach provides a blueprint towards specializing foundation medical VLMs for real-world clinical tasks.

Problem

• Improving vision-language models for retinal image analysis
• Enhancing clinical decision-making in age-related macular degeneration
• Developing specialized curricula for medical vision-language training

Innovation

• Curriculum-based VLM specialization
• Enhanced retinal image analysis
• Outperforms general medical VLMs