🤖 AI Summary
This work addresses the lack of instance-level criteria for determining when noisy labels generated by vision-language models (VLMs) in weakly supervised learning transition from beneficial to detrimental. It introduces the first instance-level calibration method grounded in classical label noise theory, adapted for modern foundation models. By calibrating BiomedCLIP-generated weak labels using only 10–20 gold-standard annotations on three medical imaging benchmarks—PCAM, ISIC, and NIH-CXR—the study identifies performance crossover points (approximately 100, 20–50, and 250–500 samples, respectively), beyond which downstream AUC drops by up to 0.10. The findings reveal that the labeler, rather than the student model, dominates performance, and propose label space projection as a theoretically motivated improvement. The resulting decision rule demonstrates robustness across diverse pretrained architectures.
📝 Abstract
Classical noisy-label theory predicts that downstream performance under weak supervision is bounded above by the labeler's accuracy, implying a sharp crossover: once a gold-trained classifier matches the labeler, weak labels stop helping and start hurting. The prediction is theoretical; what is missing is a benchmark calibration that turns it into an instance-level statement for modern foundation-model labelers. We provide such a calibration for BiomedCLIP-generated weak labels on three medical-imaging benchmarks (PCAM, ISIC, NIH-CXR) and six downstream architectures spanning an 11x parameter range. The crossover predicted by theory appears at ng~100 on PCAM, 20-50 on ISIC, and 250-500 on NIH-CXR; weak labels above the crossover degrade AUC by up to -0.10. The location is architecture-invariant for four of five pretrained architectures, and a within-family DenseNet sweep (2.5x parameters, identical pretraining) supports the view that the labeler, not the student, is the dominant constraint. The calibration in turn produces a decision rule operable from 10-20 gold labels: compare gold-only AUC to VLM accuracy on the user's gold set. A structured-vs-random noise sign flip on NIH-CXR shows that the rate-only formulation of the bound is incomplete and identifies a concrete refinement (label-space projection) that future benchmarks can be designed to test.