TokenMinds: Pretrained User Tokens and Embeddings for User Understanding in Large Recommender Systems

2026-06-23

AI Summary

This work addresses the limitations of conventional recommendation systems, where fixed-dimensional dense user embeddings exhibit constrained representational capacity, and existing large language model (LLM)-based textual user tokens struggle to align with item attributes and model deep behavioral sequences. To overcome these challenges, the authors propose TokenMinds, a novel system that introduces semantic IDs (SIDs)—discrete representations—into user modeling for the first time. TokenMinds employs a pretrained LLM-based encoder-decoder architecture to jointly generate interpretable SID user tokens and dense embeddings, effectively balancing semantic expressiveness with compatibility for downstream recommendation tasks while enabling unified cross-scenario modeling. The system has been fully deployed across multiple YouTube production environments, demonstrating significant improvements in ranking performance and validating the effectiveness and complementary value of SID tokens in billion-scale industrial recommendation systems.

Abstract

User modeling in industrial recommender systems typically produces dense embeddings, which suffer from representational constraints inherent to fixed-dimensional vectors. An emerging alternative for discrete user representation -- using LLMs to generate text-based user tokens -- captures topical co-occurrences rather than deep sequential behavior dynamics and produces outputs that are difficult to ground to item attributes. Meanwhile, Semantic ID (SID) based item tokenization has proven effective for improving generalization in generative recommendation, yet discrete SID-based representations for users remain largely unexplored. We propose TokenMinds, an industrial-scale system that extends the PLUM framework from item retrieval to user modeling, generating both discrete SID-based user tokens and dense user embeddings via an encoder-decoder architecture adapted from pre-trained LLMs. This dual-output design provides the complementary benefits of discrete, semantically grounded user representations while maintaining compatibility with existing downstream models that rely on dense embeddings. Additionally, the shared SID vocabulary naturally extends to cross-scenario modeling: by unifying long-form and short-form video behaviors into a single model, we substantially reduce training and serving costs. We validate TokenMinds through extensive offline experiments and live launches on multiple YouTube surfaces, served on full user traffic (billions of users) via an asynchronous infrastructure that decouples representation generation from downstream scoring. Focusing on ranking as the primary downstream use case, our results confirm the practical viability of SID-based user tokens at industrial scale and demonstrate that tokens and dense embeddings provide complementary value across different production ranking systems.

Problem

• user modeling
• dense embeddings
• discrete representations
• Semantic ID
• recommender systems

Innovation

• Semantic ID
• discrete user tokens
• user modeling
• large language models
• cross-scenario recommendation