A Clinician-Centered Pipeline for Annotation and Evaluation in Ultrasound AI Studies

2026-06-17

AI Summary

This study addresses the lack of support for blinded and reproducible evaluation in existing medical imaging platforms, which hinders effective validation of clinician-centered ultrasound AI systems. To overcome this limitation, the authors propose a remote annotation and evaluation pipeline tailored for clinicians, leveraging a centralized server and a lightweight web interface to enable blinded ranking, annotation, and result review without requiring local data downloads. The system uniquely integrates multi-rater participation, centralized result aggregation, and automated statistical analysis—including Spearman’s correlation coefficient, Kendall’s τ, and top-1 preference metrics—to facilitate reproducible, clinically oriented human–AI assessment. In a fetal ultrasound segmentation task, six raters with varying expertise demonstrated moderate to strong inter-rater agreement, and models refined through later-stage active learning were significantly preferred.

Abstract

Clinician-centered evaluation is critical for validating medical AI systems, especially in ultrasound imaging where quantitative metrics do not always capture clinical usability. Existing medical image platforms primarily focus on dataset labeling. They lack integrated support for blinded model comparison and reproducible evaluation workflows. We present a clinician-centered pipeline for remote annotation and evaluation in ultrasound AI studies. The proposed pipeline uses a centralized server and lightweight browser interfaces to enable clinicians to perform annotation, blinded ranking, and review without local dataset downloads. The pipeline also supports multi-rater participation, centralized result aggregation, and automated statistical analysis. We validate the pipeline in a fetal ultrasound segmentation study with six raters spanning expert, generalist, and non-expert experience levels. The system automatically generated Spearman correlation, Kendall's $τ$, and top-1 selection statistics. Results indicated moderate to strong agreement across experts and other groups. The blinded evaluation results showed a tendency for later active learning models to be preferred. These outcomes suggest that the pipeline can support clinician-centered annotation and reproducible human-\ac{AI} evaluation studies in ultrasound imaging. The proposed pipeline is available on \href{https://github.com/13204942/SonoRate}{GitHub}.

Problem

• clinician-centered evaluation
• ultrasound AI
• blinded model comparison
• reproducible evaluation
• medical image annotation

Innovation

• clinician-centered evaluation
• ultrasound AI
• blinded model comparison
• reproducible evaluation pipeline
• remote annotation