CleanPatrick: A Benchmark for Image Data Cleaning

2025-05-16

AI Summary

Existing image data cleaning benchmarks predominantly rely on synthetic noise or small-scale manual annotations, suffering from limited realism and cross-study comparability. This work introduces the first large-scale, real-world medical image cleaning benchmark, built upon the Fitzpatrick17k dataset and annotated by 933 medically trained crowdworkers who provided 496,000 binary labels. The benchmark systematically identifies off-topic samples, near-duplicate images, and fine-grained label errors, establishing high-confidence ground-truth cleaning annotations. Methodologically, we reformulate cleaning as a ranking problem and propose a novel item-response-theory-inspired annotation aggregation model, integrated with expert verification and a standardized evaluation protocol. Experiments reveal that self-supervised representations achieve state-of-the-art performance in near-duplicate detection; classical methods remain highly cost-effective for off-topic sample identification under constrained annotation budgets; yet fine-grained correction of medical labels remains a critical unsolved challenge.

Technology Category

• Computer Vision: Medical and Biological Imaging
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data integration and cleaning
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Robust machine learning depends on clean data, yet current image data cleaning benchmarks rely on synthetic noise or narrow human studies, limiting comparison and real-world relevance. We introduce CleanPatrick, the first large-scale benchmark for data cleaning in the image domain, built upon the publicly available Fitzpatrick17k dermatology dataset. We collect 496,377 binary annotations from 933 medical crowd workers, identify off-topic samples (4%), near-duplicates (21%), and label errors (22%), and employ an aggregation model inspired by item-response theory followed by expert review to derive high-quality ground truth. CleanPatrick formalizes issue detection as a ranking task and adopts typical ranking metrics mirroring real audit workflows. Benchmarking classical anomaly detectors, perceptual hashing, SSIM, Confident Learning, NoiseRank, and SelfClean, we find that, on CleanPatrick, self-supervised representations excel at near-duplicate detection, classical methods achieve competitive off-topic detection under constrained review budgets, and label-error detection remains an open challenge for fine-grained medical classification. By releasing both the dataset and the evaluation framework, CleanPatrick enables a systematic comparison of image-cleaning strategies and paves the way for more reliable data-centric artificial intelligence.

Problem

• Lack of large-scale benchmarks for real-world image data cleaning
• Need for accurate detection of off-topic, near-duplicate, and label-error samples
• Challenges in evaluating cleaning methods for fine-grained medical classification

Innovation

• Large-scale benchmark using Fitzpatrick17k dataset
• Item-response theory for annotation aggregation
• Formalizes issue detection as ranking task