An integrated language-vision foundation model for conversational diagnostics and triaging in primary eye care

2025-05-13

AI Summary

Existing ophthalmic AI models are predominantly single-task, lacking natural language interaction capabilities and multimodal synergy. Method: We propose Meta-EyeFM—the first end-to-end fine-tunable multimodal foundation model for ophthalmology—integrating a large language model (LLM) with a vision foundation model (VFM) to support natural language–driven disease detection, severity grading, clinical sign identification, and intelligent triage. It introduces a novel task-routing mechanism that dynamically dispatches specialized visual submodels and pioneers joint LLM–VFM LoRA fine-tuning to balance medical domain expertise and conversational fluency. Results: Experiments demonstrate 100% routing accuracy; Meta-EyeFM significantly outperforms Gemini-1.5-flash and GPT-4o across disease detection (≥82.2%), severity grading (≥89%), and sign identification (≥76%), achieving improvements of +11%–43% and matching clinician-level performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Current deep learning models are mostly task specific and lack a user-friendly interface to operate. We present Meta-EyeFM, a multi-function foundation model that integrates a large language model (LLM) with vision foundation models (VFMs) for ocular disease assessment. Meta-EyeFM leverages a routing mechanism to enable accurate task-specific analysis based on text queries. Using Low Rank Adaptation, we fine-tuned our VFMs to detect ocular and systemic diseases, differentiate ocular disease severity, and identify common ocular signs. The model achieved 100% accuracy in routing fundus images to appropriate VFMs, which achieved $ge$ 82.2% accuracy in disease detection, $ge$ 89% in severity differentiation, $ge$ 76% in sign identification. Meta-EyeFM was 11% to 43% more accurate than Gemini-1.5-flash and ChatGPT-4o LMMs in detecting various eye diseases and comparable to an ophthalmologist. This system offers enhanced usability and diagnostic performance, making it a valuable decision support tool for primary eye care or an online LLM for fundus evaluation.

Problem

• Develops an integrated language-vision model for eye care diagnostics
• Improves accuracy in disease detection and severity differentiation
• Enhances usability compared to task-specific deep learning models

Innovation

• Integrated language-vision model for eye care
• Routing mechanism for task-specific image analysis
• Low Rank Adaptation fine-tuned disease detection