Stop Treating Collisions Equally: Qualification-Aware Semantic ID Learning for Recommendation at Industrial Scale

2026-02-28

AI Summary

This work addresses the semantic conflicts induced by quantization in semantic ID (SID) learning and proposes QuaSID, a novel framework that introduces, for the first time, a conflict eligibility-aware mechanism to distinguish between harmful and benign conflicts and dynamically adjust repulsion strength accordingly. The method integrates Hamming distance–guided margin repulsion, conflict-aware valid pair masking, and dual-tower contrastive learning to enable end-to-end high-quality SID learning. The proposed plug-and-play repulsion loss is generalizable across diverse SID architectures. Experiments demonstrate that QuaSID improves Top-K ranking metrics by 5.9% on public benchmarks while significantly enhancing SID diversity. In large-scale A/B tests on Kuaishou’s e-commerce platform, it achieves a 2.38% increase in GMV-S2 and up to a 6.42% boost in order volume under cold-start scenarios.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Semi-Supervised Learning
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Semantic IDs (SIDs) are compact discrete representations derived from multimodal item features, serving as a unified abstraction for ID-based and generative recommendation. However, learning high-quality SIDs remains challenging due to two issues. (1) Collision problem: the quantized token space is prone to collisions, in which semantically distinct items are assigned identical or overly similar SID compositions, resulting in semantic entanglement. (2) Collision-signal heterogeneity: collisions are not uniformly harmful. Some reflect genuine conflicts between semantically unrelated items, while others stem from benign redundancy or systematic data effects. To address these challenges, we propose Qualification-Aware Semantic ID Learning (QuaSID), an end-to-end framework that learns collision-qualified SIDs by selectively repelling qualified conflict pairs and scaling the repulsion strength by collision severity. QuaSID consists of two mechanisms: Hamming-guided Margin Repulsion, which translates low-Hamming SID overlaps into explicit, severity-scaled geometric constraints on the encoder space; and Conflict-Aware Valid Pair Masking, which masks protocol-induced benign overlaps to denoise repulsion supervision. In addition, QuaSID incorporates a dual-tower contrastive objective to inject collaborative signals into tokenization. Experiments on public benchmarks and industrial data validate QuaSID. On public datasets, QuaSID consistently outperforms strong baselines, improving top-K ranking quality by 5.9% over the best baseline while increasing SID composition diversity. In an online A/B test on Kuaishou e-commerce with a 5% traffic split, QuaSID increases ranking GMV-S2 by 2.38% and improves completed orders on cold-start retrieval by up to 6.42%. Finally, we show that the proposed repulsion loss is plug-and-play and enhances a range of SID learning frameworks across datasets.

Problem

• Semantic ID
• collision problem
• collision-signal heterogeneity
• recommendation
• industrial scale

Innovation

• Semantic ID
• Collision-aware learning
• Hamming-guided repulsion
• Conflict-aware masking
• Contrastive tokenization