When Repository Labels Are Not Image-Level Truth: A Supervision Auditing Framework for Chest Radiograph AI

2026-08-10

AI Summary

This study addresses the widespread issue of label inaccuracies in public chest X-ray datasets, where annotations derived from radiology reports often misrepresent actual pathological findings, thereby compromising model training with unreliable supervisory signals. To mitigate this, the authors propose the Repository Supervision Auditing (RSA) framework, which employs expert image-level annotations to audit label consistency prior to model development, identify systematic biases, and construct a verified evaluation cohort. Their analysis reveals severe discrepancies between report-based labels and radiological evidence: only 1% of cases with expert-confirmed cardiomegaly were correctly labeled, while nearly half were erroneously annotated as “no abnormality.” A DenseNet121 model trained on the corrected cohort achieved a test ROC-AUC of 0.853, demonstrating that supervision auditing is a critical prerequisite for robust medical imaging AI development.

Abstract

Public chest X-ray repositories are widely used to train medical AI systems, yet their labels are typically extracted from radiology reports rather than verified directly on images. As a result, repository labels are often treated as image-level ground truth without validating whether they reflect what is actually visible in the radiograph. We introduce Repository Supervision Auditing (RSA), a framework that evaluates repository-derived labels against expert image-level annotations before model development. Using cardiomegaly in MIMIC-CXR as a case study, RSA compares repository labels with radiologist-reviewed image annotations, characterizes disagreement sources, and builds a curated cohort for deployment-oriented evaluation. Repository-derived cardiomegaly labels showed near-zero agreement with expert image-level assessment, identifying only 1% of expert-confirmed cases. Most discrepancies resulted from non-mention rather than explicit report negation, with expert-confirmed cardiomegaly identified in nearly half of studies assigned a repository-derived No Finding label. Using the resulting expert-curated cohort, a DenseNet121 model achieved a test ROC-AUC of 0.853. These findings show that repository labels may not reliably represent image-level truth and highlight supervision auditing as a critical step for developing trustworthy medical imaging AI.

Problem

• chest radiograph
• repository labels
• image-level truth
• supervision auditing
• medical AI

Innovation

• supervision auditing
• chest radiograph
• label validation
• medical AI
• image-level annotation