Benchmarking non-conformity score functions in conformal prediction

2026-05-24

AI Summary

This study addresses the lack of systematic evaluation of non-conformity scoring functions in conformal prediction, particularly their inability to simultaneously balance prediction set size and coverage accuracy under class-imbalanced conditions. We conduct a comprehensive comparison of mainstream scoring functions, propose a novel variant, and introduce, for the first time, a unified mechanism for evaluating prediction set size. Within the conformal prediction framework, we empirically analyze the performance of these scoring functions on both standard and imbalanced datasets, elucidating their impact on prediction set characteristics. Experimental results demonstrate that the proposed method significantly reduces prediction set size while maintaining valid coverage, with especially pronounced improvements in class-imbalanced tasks.

Abstract

Conformal prediction is a useful and versatile alternative to model calibration in machine learning classification. It replaces single-class prediction with prediction sets, guaranteeing that the \textit{a priori} probability of the prediction sets containing the true class is larger than or equal to a pre-specified rate. The size and usefulness of the prediction sets relies heavily on the choice of the non-conformity score function. The scientific literature contains many examples of non-conformity score functions but there is an absence of studies examining their properties and effectiveness. In this paper, we give an overview of properties of non-conformity score functions. We give examples of non-conformity score functions in the existing literature and introduce original modifications. We introduce an original method of evaluating the prediction set sizes of conformal predictors and use it to provide a comparison between non-conformity score functions. We also examine efficacy of different non-conformity score functions for class-conditional conformal prediction in a setting with imbalanced classes.

Problem

• conformal prediction
• non-conformity score
• prediction sets
• class imbalance
• score function evaluation

Innovation

• conformal prediction
• non-conformity score
• prediction sets
• class-conditional coverage
• imbalanced classes