Enhancing Embedding Representation Stability in Recommendation Systems with Semantic ID

2025-04-02

AI Summary

Online recommendation systems suffer from embedding instability and representation drift due to explosive ID cardinality, dynamic ID evolution, and long-tailed distributions. To address these challenges, we propose Semantic ID Prefix n-gram: a content-based hierarchical clustering approach that groups IDs into semantically coherent buckets—replacing conventional random hashing—and introduces ID-space parameterization with attention-based adaptation to enhance model robustness across ID lifecycle stages (e.g., cold-start and decay). Deployed in Meta’s ad ranking system, our method significantly improves prediction stability, yielding measurable gains in AUC and CTR, while boosting conversion rates for tail IDs by 12.7%—without incurring additional online latency. The core contribution is the first semantic-aware ID tokenization framework that jointly optimizes generalizability and embedding stability, effectively mitigating long-tail modeling difficulties and embedding drift.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Recommender Systems

Application Category

• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

The exponential growth of online content has posed significant challenges to ID-based models in industrial recommendation systems, ranging from extremely high cardinality and dynamically growing ID space, to highly skewed engagement distributions, to prediction instability as a result of natural id life cycles (e.g, the birth of new IDs and retirement of old IDs). To address these issues, many systems rely on random hashing to handle the id space and control the corresponding model parameters (i.e embedding table). However, this approach introduces data pollution from multiple ids sharing the same embedding, leading to degraded model performance and embedding representation instability. This paper examines these challenges and introduces Semantic ID prefix ngram, a novel token parameterization technique that significantly improves the performance of the original Semantic ID. Semantic ID prefix ngram creates semantically meaningful collisions by hierarchically clustering items based on their content embeddings, as opposed to random assignments. Through extensive experimentation, we demonstrate that Semantic ID prefix ngram not only addresses embedding instability but also significantly improves tail id modeling, reduces overfitting, and mitigates representation shifts. We further highlight the advantages of Semantic ID prefix ngram in attention-based models that contextualize user histories, showing substantial performance improvements. We also report our experience of integrating Semantic ID into Meta production Ads Ranking system, leading to notable performance gains and enhanced prediction stability in live deployments.

Problem

• Addresses embedding instability in recommendation systems
• Improves tail ID modeling and reduces overfitting
• Enhances prediction stability in dynamic ID spaces

Innovation

• Semantic ID prefix ngram token parameterization
• Hierarchical clustering for semantic collisions
• Improved tail id modeling and stability