A multimodal AI agent for clinical decision support in ophthalmology

2025-11-12

AI Summary

Current ophthalmic AI systems suffer from limited flexibility, poor interpretability, and inadequate multimodal integration capabilities. To address these challenges, we propose EyeAgent—the first clinical decision-support intelligent agent framework for ophthalmology, built upon the DeepSeek-V3 large language model. EyeAgent dynamically orchestrates 53 validated, ophthalmology-specific tools across 23 imaging modalities, enabling coordinated analysis for classification, segmentation, detection, and structured report generation. Its modular, interpretable architecture ensures tight alignment with clinical workflows, supporting stepwise reasoning and human-AI collaboration. Evaluated on 200 real-world clinical cases, EyeAgent achieves a 93.7% tool-selection accuracy and receives expert ratings exceeding 88% across usability and reliability metrics. When deployed as a diagnostic aid, it improves overall diagnostic accuracy by 18.51% and enhances report quality by 19%, demonstrating significant clinical utility and robust multimodal reasoning capability.

Technology Category

• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Artificial intelligence has shown promise in medical imaging, yet most existing systems lack flexibility, interpretability, and adaptability - challenges especially pronounced in ophthalmology, where diverse imaging modalities are essential. We present EyeAgent, the first agentic AI framework for comprehensive and interpretable clinical decision support in ophthalmology. Using a large language model (DeepSeek-V3) as its central reasoning engine, EyeAgent interprets user queries and dynamically orchestrates 53 validated ophthalmic tools across 23 imaging modalities for diverse tasks including classification, segmentation, detection, image/report generation, and quantitative analysis. Stepwise ablation analysis demonstrated a progressive improvement in diagnostic accuracy, rising from a baseline of 69.71% (using only 5 general tools) to 80.79% when the full suite of 53 specialized tools was integrated. In an expert rating study on 200 real-world clinical cases, EyeAgent achieved 93.7% tool selection accuracy and received expert ratings of more than 88% across accuracy, completeness, safety, reasoning, and interpretability. In human-AI collaboration, EyeAgent matched or exceeded the performance of senior ophthalmologists and, when used as an assistant, improved overall diagnostic accuracy by 18.51% and report quality scores by 19%, with the greatest benefit observed among junior ophthalmologists. These findings establish EyeAgent as a scalable and trustworthy AI framework for ophthalmology and provide a blueprint for modular, multimodal, and clinically aligned next-generation AI systems.

Problem

• Developing a flexible AI framework for ophthalmology clinical decision support
• Integrating multiple imaging modalities and tools for comprehensive diagnosis
• Improving diagnostic accuracy and interpretability in ophthalmology AI systems

Innovation

• Uses large language model as reasoning engine
• Dynamically orchestrates 53 specialized ophthalmic tools
• Integrates multimodal imaging across 23 modalities