Split Conformal Classification with Unsupervised Calibration

2025-10-08

AI Summary

Existing split conformal prediction requires a separate labeled calibration set, preventing full utilization of labeled samples for training and incurring substantial additional annotation costs. This work addresses classification tasks and proposes, for the first time, an unsupervised calibration-based split conformal prediction framework that eliminates dependence on labeled calibration data. Our method constructs the nonconformity score distribution solely from unlabeled data and jointly optimizes the supervised training objective to produce set-valued predictions with controlled probabilistic coverage. Theoretical analysis establishes finite-sample coverage deviation guarantees. Empirical evaluation demonstrates that, when calibrated exclusively on unlabeled data, our approach achieves coverage performance comparable to supervised baselines, incurs only moderate computational overhead, significantly improves labeled-data utilization efficiency, and lowers practical deployment barriers.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance

Abstract

Methods for split conformal prediction leverage calibration samples to transform any prediction rule into a set-prediction rule that complies with a target coverage probability. Existing methods provide remarkably strong performance guarantees with minimal computational costs. However, they require to use calibration samples composed by labeled examples different to those used for training. This requirement can be highly inconvenient, as it prevents the use of all labeled examples for training and may require acquiring additional labels solely for calibration. This paper presents an effective methodology for split conformal prediction with unsupervised calibration for classification tasks. In the proposed approach, set-prediction rules are obtained using unsupervised calibration samples together with supervised training samples previously used to learn the classification rule. Theoretical and experimental results show that the presented methods can achieve performance comparable to that with supervised calibration, at the expenses of a moderate degradation in performance guarantees and computational efficiency.

Problem

• Eliminates need for labeled calibration data in conformal prediction
• Enables using all labeled examples for training classification models
• Maintains performance comparable to supervised calibration methods

Innovation

• Uses unsupervised calibration samples for prediction
• Combines unsupervised and supervised training samples
• Achieves comparable performance with supervised calibration