PIT: A Dynamic Personalized Item Tokenizer for End-to-End Generative Recommendation

2026-02-09

AI Summary

This work addresses a critical limitation in existing generative recommender systems, which rely on static and decoupled item tokenization schemes that ignore collaborative signals, thereby hindering end-to-end co-evolution between item indexing and the recommendation model. To overcome this, we propose the PIT framework, which introduces a novel co-generative architecture that jointly trains a dynamic, personalized item tokenizer and a generative recommender, enabling their co-evolution under aligned collaborative signals. Furthermore, we incorporate a one-to-many beam indexing mechanism to enhance scalability and robustness for industrial deployment. Extensive experiments demonstrate that PIT significantly outperforms baseline methods across multiple real-world datasets, and large-scale A/B testing on Kuaishou shows a 0.402% increase in user app dwell time.

Technology Category

• Natural Language Processing: Generation
• Data Mining & Knowledge Management: Recommender Systems
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Generative Recommendation has revolutionized recommender systems by reformulating retrieval as a sequence generation task over discrete item identifiers. Despite the progress, existing approaches typically rely on static, decoupled tokenization that ignores collaborative signals. While recent methods attempt to integrate collaborative signals into item identifiers either during index construction or through end-to-end modeling, they encounter significant challenges in real-world production environments. Specifically, the volatility of collaborative signals leads to unstable tokenization, and current end-to-end strategies often devolve into suboptimal two-stage training rather than achieving true co-evolution. To bridge this gap, we propose PIT, a dynamic Personalized Item Tokenizer framework for end-to-end generative recommendation, which employs a co-generative architecture that harmonizes collaborative patterns through collaborative signal alignment and synchronizes item tokenizer with generative recommender via a co-evolution learning. This enables the dynamic, joint, end-to-end evolution of both index construction and recommendation. Furthermore, a one-to-many beam index ensures scalability and robustness, facilitating seamless integration into large-scale industrial deployments. Extensive experiments on real-world datasets demonstrate that PIT consistently outperforms competitive baselines. In a large-scale deployment at Kuaishou, an online A/B test yielded a substantial 0.402% uplift in App Stay Time, validating the framework's effectiveness in dynamic industrial environments.

Problem

• Generative Recommendation
• Item Tokenization
• Collaborative Signals
• End-to-End Learning
• Dynamic Tokenizer

Innovation

• generative recommendation
• dynamic tokenization
• collaborative signal alignment
• co-evolution learning
• end-to-end training