Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning

2026-07-15

AI Summary

This work addresses the challenge of directly optimizing long-term user retention in recommender systems, where signals are sparse, delayed, and difficult to attribute. The authors propose a model-agnostic downstream reward learning framework that constructs a unified and generalizable proxy reward by offline selection of early-observable, highly predictive multi-source user behaviors. This proxy reward is seamlessly integrated into ranking models to optimize long-term user value, circumventing the need for task-specific reward engineering or complex sequential modeling. Evaluated through extensive A/B experiments across multiple core scenarios at Pinterest, the approach significantly improves user engagement and retention metrics and has been successfully deployed in production.

Abstract

As recommender systems mature in the past few years, their optimization objectives have evolved from a primary focusing on short-term behavioral signals to a broader emphasis on long-term user engagement and retention. However, directly optimizing retention is difficult because return signals are sparse, delayed, and only partially attributable to earlier recommendations. Prior work has addressed this challenge with sequential modeling and reinforcement learning, but these approaches typically require task specific reward engineering, substantial computational overhead, and surface specific implementations that are difficult to generalize. In this paper, we present a unified, model-agnostic downstream reward framework for optimizing long-term user value in large-scale recommendation systems. First, we formulate the downstream reward learning problem and develop an offline screening framework to identify session level behaviors that are both observable early and predictive of future retention. We then propose several model-agnostic downstream rewards signals derived from observed user action patterns across multiple sources. We further discuss the engineering effort to productionize the proposed rewards derivations and challenges we faced when adding them to our ranking models. Online A/B experiments demonstrate consistent improvements in engagement and retention-related metrics, and the framework has been deployed across multiple Pinterest surfaces, including Homefeed, Related Pins, Search, and Notifications.

Problem

• long-term user engagement
• user retention
• downstream rewards
• recommendation systems
• sparse delayed signals

Innovation

• model-agnostic
• downstream reward learning
• long-term user engagement
• offline screening
• recommendation systems