Modeling Cascaded Delay Feedback for Online Net Conversion Rate Prediction: Benchmark, Insights and Solutions

2026-01-27

AI Summary

This work addresses the limitation of traditional conversion rate (CVR) prediction, which overlooks user refund behavior and thus fails to accurately reflect user satisfaction and business value. To tackle the cascaded delayed feedback inherent in net conversion rate (NetCVR) prediction—spanning click, conversion, and refund stages—the authors propose TESLA, a novel framework that jointly models conversion and refund through cascaded structures, incorporates stage-wise debiasing, employs delay-aware ranking loss, and leverages an online continual learning mechanism to effectively handle the opposing delay effects across the two stages. The study also introduces CASCADE, the first large-scale open-source NetCVR dataset, revealing key characteristics such as strong temporal dynamics. Experimental results demonstrate that TESLA significantly outperforms state-of-the-art methods on CASCADE, achieving relative improvements of 12.41% in RI-AUC and 14.94% in RI-PRAUC.

Technology Category

• Machine Learning: Online Learning & Bandits
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

In industrial recommender systems, conversion rate (CVR) is widely used for traffic allocation, but it fails to fully reflect recommendation effectiveness because it ignores refund behavior. To better capture true user satisfaction and business value, net conversion rate (NetCVR), defined as the probability that a clicked item is purchased and not refunded, has been proposed.Unlike CVR, NetCVR prediction involves a more complex multi-stage cascaded delayed feedback process. The two cascaded delays from click to conversion and from conversion to refund have opposite effects, making traditional CVR modeling methods inapplicable. Moreover, the lack of open-source datasets and online continuous training schemes further hinders progress in this area.To address these challenges, we introduce CASCADE (Cascaded Sequences of Conversion and Delayed Refund), the first large-scale open dataset derived from the Taobao app for online continuous NetCVR prediction. Through an in-depth analysis of CASCADE, we identify three key insights: (1) NetCVR exhibits strong temporal dynamics, necessitating online continuous modeling; (2) cascaded modeling of CVR and refund rate outperforms direct NetCVR modeling; and (3) delay time, which correlates with both CVR and refund rate, is an important feature for NetCVR prediction.Based on these insights, we propose TESLA, a continuous NetCVR modeling framework featuring a CVR-refund-rate cascaded architecture, stage-wise debiasing, and a delay-time-aware ranking loss. Extensive experiments demonstrate that TESLA consistently outperforms state-of-the-art methods on CASCADE, achieving absolute improvements of 12.41 percent in RI-AUC and 14.94 percent in RI-PRAUC on NetCVR prediction. The code and dataset are publicly available at https://github.com/alimama-tech/NetCVR.

Problem

• NetCVR
• cascaded delayed feedback
• online conversion rate prediction
• refund behavior
• industrial recommender systems

Innovation

• NetCVR
• cascaded delayed feedback
• online continuous learning
• delay-aware modeling
• refund rate prediction