Benchmarking Instance-Dependent Label Noise with Controlled Corruptions

2026-06-12

AI Summary

This work addresses the limitations of existing instance-dependent label noise (IDN) benchmarks, which typically originate from uncontrolled, imperfect annotators and exhibit ambiguous noise mechanisms. To overcome this, we propose the Controllable Instance-Dependent Label Noise (CILN) framework, which explicitly models IDN through controllable input perturbations combined with a diverse pool of annotators who vote on labels for perturbed samples. This approach enables precise control over both the structure and intensity of label noise. Using CILN, we construct a comprehensive benchmark comprising 90 combinations of perturbation types and noise levels across CIFAR-10, MNIST, and Adult datasets, offering a more realistic simulation of human annotation uncertainty. Our experiments reveal that prevailing methods such as Co-Teaching and DivideMix suffer significant performance degradation under specific noise configurations, highlighting the critical influence of noise structure on algorithmic robustness.

Abstract

Synthetic instance-dependent label noise (IDN) benchmarks are widely used to evaluate noisy-label learning methods, yet existing approaches typically generate noise through imperfect annotators or classifier raters, leaving the source of ambiguity implicit. We introduce CILN, a benchmark generation framework that creates IDN through controlled input corruptions. A diverse voter pool labels corrupted instances, producing benchmark datasets in which both the source and severity of ambiguity are explicit and controllable. Using CIFAR10, MNIST, and Adult, we construct 90 benchmark settings spanning multiple corruption families and severity levels. Our experiments show that the resulting benchmarks exhibit genuine instance-dependent noise, provide diverse confusion structures, and, on CIFAR-10, can produce label distributions that are closer to human uncertainty than an existing synthetic IDN benchmark. We further demonstrate that corruption-mediated IDN can expose failure modes of popular noisy-label learning methods, including Co-Teaching and DivideMix, that are not observed under comparable levels of rater-fallibility noise. These findings suggest that noise structure, not only noise rate, plays an important role in benchmark difficulty and algorithm behavior. By making ambiguity generation explicit and controllable, CILN provides a complementary benchmarking framework for studying noisy-label learning under diverse sources of instance difficulty.

Problem

• instance-dependent label noise
• benchmarking
• controlled corruptions
• noisy-label learning
• ambiguity source

Innovation

• instance-dependent label noise
• controlled corruptions
• benchmark generation
• noisy-label learning
• ambiguity control