Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score

2026-06-23

AI Summary

This work addresses the need for uncertainty quantification in ordinal classification within high-stakes domains such as medicine and finance, where errors of varying severity must be rigorously controlled. Existing conformal prediction methods are limited by their choice of nonconformity functions, which often fail to reflect the inherent ordering of classes. To overcome this, the authors propose a novel conformal prediction approach based on the Ranked Probability Score (RPS), introducing RPS as a natural nonconformity measure that captures ordinal risk. This method yields continuous prediction sets centered around the median, avoids greedy search procedures, and maintains model-agnosticism and computational efficiency. It is applicable to both evaluation-based and grouping-based ordinal tasks. Empirical results across multiple image and tabular ordinal datasets demonstrate that the proposed method achieves a superior trade-off between prediction set width and the severity of miscoverage compared to existing approaches.

Abstract

Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction (CP) provides distribution-free prediction sets with marginal coverage guarantees; however, its practical effectiveness depends critically on the choice of nonconformity function. We introduce a CP method for ordinal classification based on the ranked probability score (RPS), a proper scoring rule defined over cumulative predictive distributions. Although it reflects ordinal risk quite naturally, it has largely been neglected in conformal ordinal prediction (COP). When used as a measure of nonconformity, RPS yields median-centered contiguous prediction sets by construction. The method is model-agnostic, supports both assessed and grouped ordered categorical outcomes, and permits efficient implementation compared to greedy interval selection procedures. Across multiple ordinal image and tabular datasets, RPS-based CP produces contiguous prediction sets and strikes a favorable balance between prediction set width and the magnitude of ordinal miscoverage relative to existing CP methods.

Problem

• ordinal classification
• conformal prediction
• ranked probability score
• uncertainty quantification
• prediction sets