Post-Calibration Reliability Reranking of Relevance Decisions via Label-wise Monotone Projection

2026-08-10

AI Summary

Existing post-hoc calibration methods still exhibit label-dependent reliability discrepancies at the same confidence level, often leading systems to overtrust incorrect predictions or erroneously discard correct ones. This work proposes Label-level Monotonic Reliability Projection (MRP), which learns label-specific monotonic functions to map calibrated confidences to more reliable correctness indicators while preserving the original predicted labels and class probabilities. By leveraging these refined reliability estimates, MRP re-ranks fixed predictions according to residual risk. Introducing, for the first time, label-level modeling of residual reliability, MRP achieves superior performance compared to global confidence remapping through fine-grained monotonic projections. Evaluated on six information access relevance datasets, MRP significantly improves reranking effectiveness and fallback utility while maintaining full-coverage accuracy and expected calibration error (ECE) unchanged.

Abstract

Web search, product search, and question-answering retrieval systems often assign a relevance label and confidence score to each query-candidate pair. The relevance label describes how well a page, product, or passage matches the query, while the confidence often guides downstream use or fallback decisions. Post-hoc calibration is therefore needed because misaligned confidence can make systems over-trust wrong predictions or unnecessarily defer correct ones. However, calibration mainly aligns confidence with average correctness, and does not remove predicted-label-dependent reliability differences that remain within the same calibrated confidence level. We address this gap with Label-wise Monotone Reliability Projection (MRP), which learns label-wise monotone functions that map calibrated confidence to correctness reliability while preserving the original predicted labels and class probabilities. The resulting reliability score reranks fixed predictions according to residual risk. Across six information access relevance datasets and multiple post-hoc calibrators, MRP improves reliability reranking and average fallback utility while preserving full-coverage accuracy and ECE. Structural ablations show that the main gains come from label-wise residual reliability rather than from global confidence remapping. We further analyze when MRP reliability scores can be embedded back into top-label probability geometry, showing that this projection is useful as a compatibility analysis but is distinct from the main reliability-reranking objective. The implementation will be made publicly available.

Problem

• post-hoc calibration
• reliability reranking
• label-wise reliability
• confidence calibration
• relevance prediction

Innovation

• Monotone Reliability Projection
• Post-hoc Calibration
• Label-wise Reliability
• Confidence Reranking
• Fallback Utility