Break the Inaccessible Boundary: Distilling Post-Conversion Content for User Retention Modeling

2026-04-28

AI Summary

This work addresses the challenge of user retention prediction in real-time bidding advertising systems, where informative post-conversion onboarding content is unavailable during inference, leading to feature leakage and a train-inference discrepancy if naively incorporated. To bridge this gap, the authors propose OCARM, a novel framework that introduces knowledge distillation into this setting for the first time. In the first stage, a hierarchical teacher encoder is trained using post-conversion onboarding content; in the second stage, a student encoder—restricted to observable pre-conversion features—learns to approximate the teacher’s representations through a two-stage alignment distillation process, thereby implicitly capturing future signals. Extensive experiments demonstrate that OCARM significantly improves retention prediction performance in both offline evaluation and online A/B tests, effectively mitigating the mismatch between training and inference.

Abstract

User retention is a key metric to measure long-term engagement in modern platforms. In real-time bidding (RTB) advertising system for user re-engagement, the retention model is required to predict future revisit probability at bidding time, before the user converts and consumes any content. Although post-conversion content, termed Onboarding Content, provides highly informative signals for retention prediction, directly using it in training causes severe feature leakage and creates a gap between training and serving. To address this issue, we propose OCARM, a two-stage distillation-aligned framework for Onboarding Content Augmented Retention Modeling, enabling the model to implicitly capture future content using only observable features during inference. In the first stage, we deliberately expose onboarding content to train a hierarchical encoder that produces teacher representations. In the second stage, a user encoder is aligned with the frozen teacher through distillation, allowing the model to approximate the inaccessible onboarding signals without leakage. Extensive offline experiments and online A/B tests demonstrate that our framework achieves consistent improvements in a real-world growth scenario.

Problem

• user retention
• feature leakage
• real-time bidding
• onboarding content
• training-serving gap

Innovation

• knowledge distillation
• user retention modeling
• onboarding content
• feature leakage
• two-stage framework