Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems

2025-04-19

AI Summary

Existing calibration methods for deep ranking models rely on predefined transformation functions (e.g., piecewise-linear), limiting flexibility and expressive capacity in calibrating absolute prediction values. Method: We propose an end-to-end calibration framework based on Unconstrained Monotonic Neural Networks (UMNN)—the first application of UMNN to ranking calibration—combined with a novel Smooth Calibration Loss (SCLoss). This approach enforces strict monotonicity while modeling arbitrarily complex monotonic mappings, eliminating restrictive functional-form assumptions. Contribution/Results: Our method simultaneously preserves ranking order and achieves high representational power. Offline experiments demonstrate significant improvements in calibration accuracy. Deployed in Kuaishou’s large-scale video ranking system, it yields statistically significant gains in key metrics, including click-through rate (CTR) and average watch time per user.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Nonmonotonic Reasoning
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Ranking models primarily focus on modeling the relative order of predictions while often neglecting the significance of the accuracy of their absolute values. However, accurate absolute values are essential for certain downstream tasks, necessitating the calibration of the original predictions. To address this, existing calibration approaches typically employ predefined transformation functions with order-preserving properties to adjust the original predictions. Unfortunately, these functions often adhere to fixed forms, such as piece-wise linear functions, which exhibit limited expressiveness and flexibility, thereby constraining their effectiveness in complex calibration scenarios. To mitigate this issue, we propose implementing a calibrator using an Unconstrained Monotonic Neural Network (UMNN), which can learn arbitrary monotonic functions with great modeling power. This approach significantly relaxes the constraints on the calibrator, improving its flexibility and expressiveness while avoiding excessively distorting the original predictions by requiring monotonicity. Furthermore, to optimize this highly flexible network for calibration, we introduce a novel additional loss function termed Smooth Calibration Loss (SCLoss), which aims to fulfill a necessary condition for achieving the ideal calibration state. Extensive offline experiments confirm the effectiveness of our method in achieving superior calibration performance. Moreover, deployment in Kuaishou's large-scale online video ranking system demonstrates that the method's calibration improvements translate into enhanced business metrics. The source code is available at https://github.com/baiyimeng/UMC.

Problem

• Calibrating absolute values in deep ranking systems
• Overcoming inflexibility of predefined calibration functions
• Enhancing calibration with unconstrained monotonic neural networks

Innovation

• Unconstrained Monotonic Neural Network for calibration
• Smooth Calibration Loss optimizes flexible networks
• Enhances ranking system accuracy and business metrics