Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shifts with Unlabeled Data

2025-07-14

AI Summary

Conformal prediction fails to guarantee valid coverage under non-exchangeable distribution shifts, such as covariate shift, label shift, and concept drift. Method: We propose a robust calibration framework that requires no prior assumption on the shift type. Leveraging optimal transport theory—introduced to conformal prediction for the first time—we estimate the distributional divergence between unlabeled calibration and test sets, and dynamically adjust the score threshold in split conformal prediction to adaptively compensate for coverage loss. Contribution/Results: Our method operates solely on unlabeled data and provides universal mitigation against arbitrary non-exchangeable shifts. Experiments across diverse shift scenarios demonstrate near-ideal coverage (e.g., 89.7% ± 0.3% achieved for a 90% target), significantly outperforming existing calibration approaches while maintaining both statistical validity and predictive set compactness.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

Conformal prediction is a distribution-free uncertainty quantification method that has gained popularity in the machine learning community due to its finite-sample guarantees and ease of use. Its most common variant, dubbed split conformal prediction, is also computationally efficient as it boils down to collecting statistics of the model predictions on some calibration data not yet seen by the model. Nonetheless, these guarantees only hold if the calibration and test data are exchangeable, a condition that is difficult to verify and often violated in practice due to so-called distribution shifts. The literature is rife with methods to mitigate the loss in coverage in this non-exchangeable setting, but these methods require some prior information on the type of distribution shift to be expected at test time. In this work, we study this problem via a new perspective, through the lens of optimal transport, and show that it is possible to estimate the loss in coverage and mitigate it in case of distribution shift.

Problem

• Addressing non-exchangeable data in conformal prediction
• Mitigating coverage loss due to distribution shifts
• Using optimal transport for coverage estimation and correction

Innovation

• Uses optimal transport for coverage estimation
• Mitigates distribution shifts with unlabeled data
• Enhances conformal prediction under non-exchangeability