Classification with Reject Option: Distribution-free Error Guarantees via Conformal Prediction

2025-06-26

AI Summary

Machine learning models for binary classification often yield unreliable predictions by forcing outputs even under low confidence. Method: This paper rigorously integrates conformal prediction (CP) into the reject learning framework for the first time, proposing a distribution-free rejection mechanism that abstains from prediction when confidence is insufficient—rather than issuing forced outputs. Contribution/Results: We derive provable upper bounds on the error rate and finite-sample estimators thereof, explicitly characterizing the trade-off curve between error rate and rejection rate. The tightness of these theoretical bounds is empirically validated across multiple CP settings—including full conformal, offline batch-inductive conformal, and others. Crucially, our approach imposes no assumptions on the underlying data distribution and allows users to specify an arbitrary error tolerance threshold. This work thus delivers a theoretically grounded, practically deployable rejection mechanism for high-reliability AI systems.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Machine learning (ML) models always make a prediction, even when they are likely to be wrong. This causes problems in practical applications, as we do not know if we should trust a prediction. ML with reject option addresses this issue by abstaining from making a prediction if it is likely to be incorrect. In this work, we formalise the approach to ML with reject option in binary classification, deriving theoretical guarantees on the resulting error rate. This is achieved through conformal prediction (CP), which produce prediction sets with distribution-free validity guarantees. In binary classification, CP can output prediction sets containing exactly one, two or no labels. By accepting only the singleton predictions, we turn CP into a binary classifier with reject option. Here, CP is formally put in the framework of predicting with reject option. We state and prove the resulting error rate, and give finite sample estimates. Numerical examples provide illustrations of derived error rate through several different conformal prediction settings, ranging from full conformal prediction to offline batch inductive conformal prediction. The former has a direct link to sharp validity guarantees, whereas the latter is more fuzzy in terms of validity guarantees but can be used in practice. Error-reject curves illustrate the trade-off between error rate and reject rate, and can serve to aid a user to set an acceptable error rate or reject rate in practice.

Problem

• Develops binary classification with reject option using conformal prediction
• Provides theoretical error rate guarantees for rejected predictions
• Analyzes trade-offs between error rate and reject rate practically

Innovation

• Conformal prediction for error guarantees
• Binary classification with reject option
• Error-reject curves for trade-off analysis