Metrics or Mirage? An Audit of Evaluation Inconsistencies in Colonoscopy Polyp Segmentation Benchmarks

2026-07-09

AI Summary

This study addresses the lack of standardized evaluation protocols in colonoscopic polyp segmentation, which undermines reliable model comparison. Through a systematic audit of 27 papers, we uncover three critical structural issues: the omission of boundary-sensitive metrics such as Hausdorff distance, inconsistent train/test split protocols, and the absence of statistical significance testing. To rectify these shortcomings, we establish a unified evaluation framework, re-evaluating representative models under multiple protocols with rigorous significance analysis, and propose a domain-specific five-item reporting checklist (PSRC). Our findings reveal that conventional metrics often obscure deficiencies in boundary accuracy and recall, and that the ranking of “best-performing” models varies substantially with evaluation settings—demonstrating the unreliability of current leaderboards and providing a methodological foundation for standardized polyp segmentation assessment.

Abstract

Progress in colonoscopy polyp segmentation is routinely reported through leaderboard comparisons on a small set of public benchmarks. We argue that this apparent progress is difficult to verify: a systematic audit of \textbf{27 papers} published between 2015 and 2026 reveals three structural problems in how the community evaluates models. \textbf{First}, 25 of 27 papers \textit{omit the Hausdorff distance}. Hausdorff distance is a boundary-accuracy metric with direct clinical relevance for detecting flat or small polyps, and is a standard in radiotherapy segmentation. \textbf{Second}, at least five \textit{incompatible train/test split protocols} co-exist across papers reporting results on the same two datasets (Kvasir-SEG and CVC-ClinicDB), making published Dice scores non-comparable even when they appear in the same leaderboard column. \textbf{Third}, 26 of 27 papers make \textit{performance claims without any statistical significance test}. Strikingly, four papers published \emph{after} the Metrics Reloaded framework~\cite{metricsreloaded2024} (Maier-Hein et al., \textit{Nature Methods} 2024) perpetuate these same problems, suggesting that general-purpose metric guidance has not yet reached the colonoscopy sub-community. To show these problems are not merely cosmetic, we re-evaluate five representative models under three controlled protocols with a single uniform scorer, and find that the reported metric conceals large boundary and recall failures, that the ``best'' model changes with the metric, and that near-tied rankings reverse across random splits. We propose a five-point \textbf{Polyp Segmentation Reporting Checklist}~(PSRC) as a lightweight, domain-adapted corrective.

Problem

• polyp segmentation
• evaluation inconsistency
• benchmarking
• Hausdorff distance
• statistical significance

Innovation

• polyp segmentation
• evaluation benchmark
• Hausdorff distance
• train/test split inconsistency
• statistical significance