Noise-Adaptive Conformal Classification with Marginal Coverage

2025-01-29

AI Summary

Real-world datasets commonly suffer from random label noise and violate the exchangeability assumption, undermining the validity of conventional conformal prediction. Method: This paper proposes the first adaptive conformal classification framework robust to label noise. It dynamically adjusts quantile thresholds via calibrated error modeling and noise intensity estimation, enabling robust marginal coverage control without assuming data exchangeability. Contribution/Results: Theoretically, it breaks the exchangeability constraint inherent in classical conformal inference, delivering tight marginal coverage guarantees. Empirically, on real-world noisy benchmarks—including CIFAR-10H and BigEarthNet—the method reduces average prediction set size by 18% while strictly maintaining the target coverage level (e.g., 90%). This marks the first rigorous, distribution-free conformal classifier with provable robustness to label noise and no reliance on exchangeability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Conformal inference provides a rigorous statistical framework for uncertainty quantification in machine learning, enabling well-calibrated prediction sets with precise coverage guarantees for any classification model. However, its reliance on the idealized assumption of perfect data exchangeability limits its effectiveness in the presence of real-world complications, such as low-quality labels -- a widespread issue in modern large-scale data sets. This work tackles this open problem by introducing an adaptive conformal inference method capable of efficiently handling deviations from exchangeability caused by random label noise, leading to informative prediction sets with tight marginal coverage guarantees even in those challenging scenarios. We validate our method through extensive numerical experiments demonstrating its effectiveness on synthetic and real data sets, including CIFAR-10H and BigEarthNet.

Problem

• Automatic Adjustment
• Classification Methods
• Label Noise

Innovation

• Robust Prediction
• Label Noise
• Data Imperfections