Adapting Conformal Prediction to Distribution Shifts Without Labels

2024-06-03

arXiv.org

AI Summary

To address the robustness of conformal prediction under distribution shift with unlabeled test data, this paper proposes two unsupervised adaptive methods—ECP and EACP—achieving, for the first time, distribution-shift-robust conformal inference without access to ground-truth labels. Our approach integrates model uncertainty estimation, unsupervised distribution alignment, and dynamic threshold optimization, performing calibration of the scoring function and coverage control solely on unlabeled target-domain samples during inference. Evaluated across multiple large-scale benchmarks and diverse neural network architectures, the proposed methods significantly outperform existing unsupervised baselines: they strictly maintain the target coverage level (e.g., 90%) while reducing average prediction set size to near that of supervised conformal prediction—thereby achieving a favorable trade-off between statistical validity and predictive efficiency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Conformal prediction (CP) enables machine learning models to output prediction sets with guaranteed coverage rate, assuming exchangeable data. Unfortunately, the exchangeability assumption is frequently violated due to distribution shifts in practice, and the challenge is often compounded by the lack of ground truth labels at test time. Focusing on classification in this paper, our goal is to improve the quality of CP-generated prediction sets using only unlabeled data from the test domain. This is achieved by two new methods called ECP and EACP, that adjust the score function in CP according to the base model's uncertainty on the unlabeled test data. Through extensive experiments on a number of large-scale datasets and neural network architectures, we show that our methods provide consistent improvement over existing baselines and nearly match the performance of supervised algorithms.

Problem

• Adapting prediction sets to distribution shifts without labels
• Improving conformal prediction using unlabeled test data
• Enhancing performance via model uncertainty-adjusted score functions

Innovation

• Adjusts score function using model uncertainty
• Uses unlabeled test data for adaptation
• Improves conformal prediction performance