False Confidence: Automated Labels Confound Fairness Audits in Cervical Spine Segmentation

2026-07-08

AI Summary

This study addresses bias in fairness evaluations of cervical spine MRI segmentation models stemming from reliance on machine-generated silver labels, which can lead to erroneous conclusions. For the first time, the authors conduct a fairness audit across gender, age, and race in cervical MRI segmentation and introduce the concept of “illusory confidence,” demonstrating how silver labels distort fairness assessments by compressing within-group variance rather than amplifying between-group differences. Using the CSpineSeg dataset, they compare Dice scores and fairness metrics derived from expert-annotated gold labels versus silver labels, revealing that silver labels overestimate model performance by approximately 8 Dice points and falsely elevate age-related fairness from non-significant to significant. These findings establish label source as a primary confounding factor in segmentation fairness evaluation and underscore the necessity of expert-curated gold labels.

Abstract

Automated segmentation of cervical-spine MRI is increasingly used in clinical workflows, yet no fairness audit exists for this anatomy. We show that auditing these segmentation tasks is complicated by a common property of modern segmentation datasets: expert-annotated gold labels are expensive, so abundant machine-generated (silver) labels are added to limit annotation cost. This matters because the reference used to judge a model can itself be biased. In this study, we present the first fairness audit of cervical-spine MRI segmentation across sex, age, and race using the CSpineSeg dataset. We observe that the deployed model is demographically fair, but the choice of reference label, however, is not neutral. Because a dataset's silver labels are generated by a model trained on its gold labels, any new model trained on those same gold labels agrees more with the silver labels than with expert truth: scoring identical predictions against silver rather than gold overestimates performance by ~8 Dice points and turns the fairness verdict for age from non-significant to significant - not by the gap inflation Parikh et al. report (which we term false magnitude) but by collapsing within-group variance (which we term false confidence). Reference-label provenance is thus a first-order confounder in segmentation evaluation: performance and fairness should be reported against expert labels, and any fairness claim stated together with the provenance of its reference.

Problem

• fairness audit
• cervical spine segmentation
• silver labels
• reference bias
• algorithmic fairness

Innovation

• fairness audit
• silver labels
• false confidence
• cervical spine segmentation
• reference-label provenance