EyeBench: A Call for More Rigorous Evaluation of Retinal Image Enhancement

2025-02-20

AI Summary

Current evaluation of retinal image enhancement heavily relies on pixel-level metrics (e.g., PSNR, SSIM), failing to capture clinically critical aspects such as vascular structure fidelity and diabetic retinopathy (DR) grading consistency. Moreover, existing benchmarks lack fair comparison across paired/unpaired methods and expert-driven clinical validation. Method: We propose EyeBench—the first clinically grounded, comprehensive benchmark for retinal enhancement—featuring a clinical-aligned, multi-dimensional evaluation framework that integrates downstream task performance (e.g., vessel segmentation, lesion-based DR grading), expert scoring protocols, and a newly annotated dataset. It enables joint quantitative analysis using both traditional metrics and clinical consistency measures. Contribution/Results: Systematic evaluation reveals significant semantic inconsistencies in mainstream generative models regarding clinical interpretability. EyeBench provides open-source code and a fully reproducible evaluation pipeline, establishing a foundational standard for translating retinal enhancement research into real-world clinical practice.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Over the past decade, generative models have achieved significant success in enhancement fundus images.However, the evaluation of these models still presents a considerable challenge. A comprehensive evaluation benchmark for fundus image enhancement is indispensable for three main reasons: 1) The existing denoising metrics (e.g., PSNR, SSIM) are hardly to extend to downstream real-world clinical research (e.g., Vessel morphology consistency). 2) There is a lack of comprehensive evaluation for both paired and unpaired enhancement methods, along with the need for expert protocols to accurately assess clinical value. 3) An ideal evaluation system should provide insights to inform future developments of fundus image enhancement. To this end, we propose a novel comprehensive benchmark, EyeBench, to provide insights that align enhancement models with clinical needs, offering a foundation for future work to improve the clinical relevance and applicability of generative models for fundus image enhancement. EyeBench has three appealing properties: 1) multi-dimensional clinical alignment downstream evaluation: In addition to evaluating the enhancement task, we provide several clinically significant downstream tasks for fundus images, including vessel segmentation, DR grading, denoising generalization, and lesion segmentation. 2) Medical expert-guided evaluation design: We introduce a novel dataset that promote comprehensive and fair comparisons between paired and unpaired methods and includes a manual evaluation protocol by medical experts. 3) Valuable insights: Our benchmark study provides a comprehensive and rigorous evaluation of existing methods across different downstream tasks, assisting medical experts in making informed choices. Additionally, we offer further analysis of the challenges faced by existing methods. The code is available at url{https://github.com/Retinal-Research/EyeBench}

Problem

• Develops a benchmark for retinal image enhancement evaluation
• Addresses lack of clinical relevance in existing metrics
• Provides expert-guided design for comprehensive assessment

Innovation

• Introduces multi-dimensional clinical alignment evaluation
• Develops medical expert-guided evaluation protocol
• Provides comprehensive benchmark for fundus enhancement