🤖 AI Summary
This work addresses the challenge that e-commerce ranking models often produce poorly calibrated and uninterpretable outputs, hindering reliable click-through rate (CTR) probability estimation. To resolve this, the authors propose a context-aware post-hoc calibration method that transforms model scores into well-calibrated, interpretable probabilities while preserving the original ranking order. The approach further enables stratified conditional calibration across categorical attributes such as country or device type, achieving accurate probability estimates at both global and fine-grained levels without compromising ranking performance. Experimental results on two real-world datasets demonstrate that the proposed method reduces the feature-based Expected Calibration Error (F-ECE) by over 10% compared to existing techniques, effectively meeting the practical demand in e-commerce for trustworthy probabilistic predictions.
📝 Abstract
Ranking models are extensively used in e-commerce for relevance estimation. These models often suffer from poor interpretability and no scale calibration, particularly when trained with typical ranking loss functions. This paper addresses the problem of post-hoc calibration of ranking models. We introduce MLPlatt: a simple yet effective ranking model calibration method that preserves the item ordering and converts ranker outputs to interpretable click-through rate (CTR) probabilities usable in downstream tasks. The method is context-aware by design and achieves good calibration metrics globally, and within strata corresponding to different values of a selected categorical field (such as user country or device), which is often important from a business perspective of an E-commerce platform. We demonstrate the superiority of MLPlatt over existing approaches on two datasets, achieving an improvement of over 10\% in F-ECE (Field Expected Calibration Error) compared to other methods. Most importantly, we show that high-quality calibration can be achieved without compromising the ranking quality.