An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography

2026-08-07

AI Summary

This study addresses critical limitations of existing large language models in glaucoma detection—namely hallucination, low accuracy, and inconsistent outputs—by introducing the first multi-agent collaborative framework tailored for glaucoma screening. The proposed approach employs a three-stage pipeline of assessment, tool invocation, and reflective integration, synergistically combining multimodal reasoning through language models (Gemini 2.5 Flash, GPT-5.4 mini) and specialized vision models (QAModel, SwinV2-Tiny, SegFormer-B0). This framework substantially enhances diagnostic performance: classification accuracy reaches 88%, matching that of ophthalmologists and surpassing baseline methods by 16–47 percentage points; cup-to-disc ratio estimation error is reduced by 15–50%, with correlation to expert ratings improving from weak to moderate–strong (r = 0.59–0.84); and inter-run consistency is markedly improved (κ = 0.96), effectively mitigating overdiagnosis and stochasticity inherent in single-model approaches.

Abstract

Large language models (LLMs) show promise in medical image interpretation but suffer from hallucination, limited accuracy, and run-to-run inconsistency. We developed and validated an agentic AI framework integrating LLMs with specialized deep learning tools for glaucoma detection from fundus photography. The workflow had three steps: (1) LLM initial assessment; (2) function calling to invoke specialized tools for image quality (QAModel, FundaQ-8), glaucoma classification (SwinV2-Tiny), and optic disc/cup segmentation (SegFormer-B0); and (3) LLM reflection integrating the initial impression with tool outputs. Two LLMs (Gemini 2.5 Flash, GPT-5.4 mini) were evaluated on two public datasets (ORIGA, n=100; RIM-ONE-v3, n=100) under uncropped and cropped fields of view; all images were independently graded by a masked fellowship-trained glaucoma specialist. The agentic workflow improved classification accuracy by 16 to 47 percentage points across all conditions, reaching within 6 points of the specialist; on RIM-ONE-v3 the best configurations matched the specialist accuracy of 88%. LLM-alone approaches failed in two ways: GPT-5.4 mini showed positive bias (sensitivity 95-100%, specificity 0-5%), while Gemini 2.5 Flash varied stochastically between runs; the agentic workflow corrected both. Cup-to-disc ratio error fell 15-50% (MAE 0.156-0.228 to 0.104-0.132), and correlation with specialist grading rose from weak (r=0.12-0.39) to moderate-strong (r=0.59-0.84). Run-to-run consistency rose from near-random (kappa as low as -0.01) to near-perfect (kappa up to 0.96). Integrating LLMs with specialized tools addressed key limitations of LLM-alone approaches, including over-diagnosis and run-to-run variability. Gains held for both LLMs, suggesting generalizability across backbones, and may signal a shift from monolithic models toward orchestrated multi-agent systems in medical AI.

Problem

• Large Language Models
• Glaucoma Detection
• Fundus Photography
• Hallucination
• Run-to-run Inconsistency

Innovation

• Agentic AI
• Large Language Models
• Glaucoma Detection
• Fundus Photography
• Multi-agent Framework