Transductive Conformal Inference for Ranking

2025-01-20

AI Summary

This paper addresses uncertainty quantification for black-box sorting algorithms when performing a full $(n+m)$-ranking of $n$ items with known relative order plus $m$ newly added items—without access to ground-truth joint labels. To tackle this challenge, we introduce the first conformal prediction framework for transductive full ranking, proposing a distribution-free method to construct upper bounds on unknown conformity scores via the empirical $p$-value distribution. Our approach enables label-free rank-set inference with guaranteed validity. Theoretically, it satisfies strong validity and family-wise error control (FCP) for singleton rank prediction sets. Empirically, it significantly outperforms baselines on both synthetic and real-world datasets, achieving high coverage while maintaining compact prediction set sizes.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We introduce a method based on Conformal Prediction (CP) to quantify the uncertainty of full ranking algorithms. We focus on a specific scenario where $n + m$ items are to be ranked by some ''black box'' algorithm. It is assumed that the relative (ground truth) ranking of n of them is known. The objective is then to quantify the error made by the algorithm on the ranks of the m new items among the total $(n + m)$. In such a setting, the true ranks of the n original items in the total $(n + m)$ depend on the (unknown) true ranks of the m new ones. Consequently, we have no direct access to a calibration set to apply a classical CP method. To address this challenge, we propose to construct distribution-free bounds of the unknown conformity scores using recent results on the distribution of conformal p-values. Using these scores upper bounds, we provide valid prediction sets for the rank of any item. We also control the false coverage proportion, a crucial quantity when dealing with multiple prediction sets. Finally, we empirically show on both synthetic and real data the efficiency of our CP method.

Problem

• Sorting Algorithms
• Partial Order
• Ranking Uncertainty

Innovation

• Conduction Consistency Reasoning
• Compliance Prediction
• False Coverage Ratio Control