Audited Conformal Prediction for Classification under Unknown Distribution Shift

2026-06-12

AI Summary

This work addresses the unreliability of uncertainty quantification in pretrained classification models under distribution shift by proposing Auditing Conformal Prediction (ACP). ACP leverages a small amount of labeled target-domain data to train an auxiliary auditing model that identifies samples where the original predictor is likely to fail, and incorporates this information into a conformal prediction framework. The method innovatively integrates an auditing mechanism with conformal prediction, achieving significantly improved conditional coverage while maintaining marginal coverage guarantees. Moreover, it provides explicit group-conditional coverage assurances, supported by theoretical analysis. Experimental results on both synthetic and real-world datasets demonstrate that ACP effectively balances prediction set size and conditional coverage, outperforming existing approaches.

Abstract

We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift. We propose Audited Conformal Prediction (ACP), a method that leverages a small labeled dataset from the target population to train an auxiliary audit model identifying inputs where the legacy model is likely to fail. By integrating the audit model's outputs into the conformal prediction framework, ACP produces prediction sets that guarantee marginal coverage while achieving substantially higher conditional coverage in practice than existing approaches. We develop and analyze two complementary integration strategies -- one targeting marginal coverage with improved conditional performance, the other providing explicit group-conditional coverage guarantees -- and establish theoretical guarantees for both. Experiments on synthetic and real-world datasets validate the method and illustrate trade-offs between prediction set size and conditional coverage.

Problem

• conformal prediction
• distribution shift
• uncertainty quantification
• conditional coverage
• classification

Innovation

• Audited Conformal Prediction
• distribution shift
• uncertainty quantification
• conditional coverage
• audit model