🤖 AI Summary
Complementary-label learning (CLL) has long suffered from strong theoretical assumptions and reliance on synthetic benchmarks, lacking validation on real-world data. Method: We introduce the first human-annotated real-world CLL benchmark—CLCIFAR10/20 and CLMicroImageNet10/20—featuring large-scale, high-quality complementary labels collected via a rigorous human–machine collaborative annotation protocol and stringent quality control. Contribution/Results: Empirical analysis identifies three critical bottlenecks hindering CLL’s practical deployment: annotation noise, label bias, and validation difficulty; CLL algorithms exhibit substantial performance degradation on real data. Through data-level ablation studies, we quantitatively characterize the impact of each noise source. Our benchmark establishes a new standard for robust CLL modeling and trustworthy evaluation, providing both foundational resources and methodological guidance for future research.
📝 Abstract
Complementary-label learning (CLL) is a weakly-supervised learning paradigm that aims to train a multi-class classifier using only complementary labels, which indicate classes to which an instance does not belong. Despite numerous algorithmic proposals for CLL, their practical applicability remains unverified for two reasons. Firstly, these algorithms often rely on assumptions about the generation of complementary labels, and it is not clear how far the assumptions are from reality. Secondly, their evaluation has been limited to synthetic datasets. To gain insights into the real-world performance of CLL algorithms, we developed a protocol to collect complementary labels from human annotators. Our efforts resulted in the creation of four datasets: CLCIFAR10, CLCIFAR20, CLMicroImageNet10, and CLMicroImageNet20, derived from well-known classification datasets CIFAR10, CIFAR100, and TinyImageNet200. These datasets represent the very first real-world CLL datasets. Through extensive benchmark experiments, we discovered a notable decrease in performance when transitioning from synthetic datasets to real-world datasets. We investigated the key factors contributing to the decrease with a thorough dataset-level ablation study. Our analyses highlight annotation noise as the most influential factor in the real-world datasets. In addition, we discover that the biased-nature of human-annotated complementary labels and the difficulty to validate with only complementary labels are two outstanding barriers to practical CLL. These findings suggest that the community focus more research efforts on developing CLL algorithms and validation schemes that are robust to noisy and biased complementary-label distributions.
