Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift

2026-02-16

AI Summary

This work addresses the failure of traditional conformal prediction to maintain coverage guarantees under distribution shift. Focusing on bounded label-conditional covariate shift, the authors propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and random labels, guided by classifier uncertainty. A relaxation parameter dynamically adjusts the conformity threshold to preserve a pre-specified target-domain coverage rate. By integrating domain adaptation, Wasserstein distance metrics, and conformal prediction, the method establishes a theoretical lower bound on target-domain coverage and provides a qualitative characterization of pseudo-calibration behavior. Experimental results demonstrate that the proposed approach effectively mitigates coverage degradation caused by distribution shift while maintaining reasonably sized prediction sets.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.

Problem

• conformal prediction
• distribution shift
• coverage guarantee
• pseudo-calibration
• domain adaptation

Innovation

• pseudo-calibration
• conformal prediction
• distribution shift
• coverage guarantee
• Wasserstein distance