Beyond Item Dissimilarities: Diversifying by Intent in Recommender Systems

2024-05-20

AI Summary

Long-term user experience degradation in recommender systems remains a critical challenge. Method: This paper proposes a full-page diversity optimization framework driven by cross-session stable user intents, abandoning conventional item-similarity-based diversity control. Its core innovation is the first integration of real-time dynamic intent prediction into the final ranking stage, employing a Bayesian belief updating mechanism to model and balance multi-intent representations at the intent level—not the item level—combined with serialized position-aware ranking for online intent inference. Contribution/Results: Deployed in YouTube’s live production environment, the method significantly improves DAU and user satisfaction. Empirical evidence demonstrates that intent-driven diversity effectively enhances long-term user retention and experiential enjoyment.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Recommender Systems
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Personalized content ranking and presentation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

It has become increasingly clear that recommender systems that overly focus on short-term engagement prevents users from exploring diverse interests, ultimately hurting long-term user experience. To tackle this challenge, numerous diversification algorithms have been proposed. These algorithms typically rely on measures of item similarity, aiming to maximize the dissimilarity across items in the final set of recommendations. However, in this work, we demonstrate the benefits of going beyond item-level similarities by utilizing higher-level user understanding--specifically, user intents that persist across multiple interactions--in diversification. Our approach is motivated by the observation that user behaviors on online platforms are largely driven by their underlying intents. Therefore, recommendations should ensure that diverse user intents are accurately represented. While intent has primarily been studied in the context of search, it is less clear how to incorporate real-time dynamic intent predictions into recommender systems. To address this gap, we develop a probabilistic intent-based whole-page diversification framework for the final stage of a recommender system. Starting with a prior belief of user intents, the proposed framework sequentially selects items for each position based on these beliefs and subsequently updates posterior beliefs about the intents. This approach ensures that different user intents are represented on a page, towards optimizing long-term user experience. We experiment with the intent diversification framework on YouTube, the world's largest video recommendation platform, serving billions of users daily. Live experiments on a diverse set of intents show that the proposed framework increases Daily Active Users (DAU) and overall user enjoyment, validating its effectiveness in facilitating long-term planning.

Problem

• Real-time User Preferences
• Recommendation Diversity
• Long-term User Satisfaction

Innovation

• Diversified Recommendation
• Real-time User Preference Prediction
• Enhanced User Experience