Beyond Semantic IDs: Encoding Business-Value Ranking into Document Identifiers for Generative Retrieval

2026-07-13

AI Summary

This work addresses the identifier collision problem in existing generative retrieval systems and the misalignment between their encoding objectives and business optimization goals. The authors propose Cluster-Ranked Identifier (CRID), which decouples document identifiers into two components: semantic clustering and business-value ranking. This is the first approach to explicitly embed business value into the identifier architecture, yielding a collision-free, incrementally updatable encoding structure. A complementary utility decomposition framework reveals how semantic cluster size balances personalization preferences against the generalization capacity of statistical priors. Evaluated on Taobao’s dataset comprising 300 million items, CRID outperforms the strongest embedding-based retrieval baseline in Top-K recall and achieves a 1.06% increase in gross merchandise value (GMV) upon full deployment.

Abstract

Generative Retrieval (GR) formulates retrieval as a sequence-to-sequence generation task, assigning each document a document identifier (DocID) and retrieving it through autoregressive decoding, making DocID design a critical factor in retrieval quality. However, existing schemes based on discrete representation learning suffer from inherent collision issues and create a mismatch between the DocID's encoding objective and the system's business optimization target. To address these limitations, we propose Cluster-Ranked Identifier (CRID), which decouples DocID into semantic clustering and business-value ranking, yielding collision-free identifiers that support incremental updates via intra-cluster reranking. We further introduce an analytical framework that decomposes retrieval gains into personalized preference and statistical prior generalization, revealing how semantic cluster size governs the balance between the two components. Experiments on a 300M-item Taobao e-commerce corpus show that CRID surpasses the strongest embedding-based retrieval baseline on top-K Hitrate, and delivers +1.06% GMV in full-traffic deployment.

Problem

• Generative Retrieval
• Document Identifier
• Business-Value Ranking
• Collision Issue
• Semantic Representation

Innovation

• Generative Retrieval
• Document Identifier
• Business-Value Ranking
• Semantic Clustering
• Collision-Free Encoding