How Reliable Are Semantic-ID Tokenizer Comparisons in Generative Recommendation?

2026-05-24

AI Summary

This work addresses a critical flaw in existing generative recommender systems, which erroneously equate Semantic-ID (SID) sequence matching with item recommendation during SID-level evaluation, thereby overlooking conflicts arising from multiple items sharing the same SID. Such oversights lead to substantial performance overestimation. We conduct the first systematic analysis of SID collision effects, revealing that mainstream SID tokenizers—due to feature compression—assign identical SIDs to semantically similar yet behaviorally distinct items. Across four datasets, up to 30.5% of items are affected, inflating Hit@10 by as much as 103.36%. To mitigate this bias without retraining, we propose a minimal-cost SID reallocation strategy and introduce conflict-aware, item-level evaluation metrics that effectively correct the distortion, enabling more reliable performance comparisons in generative recommendation.

Abstract

In Semantic-ID (SID) based generative recommendation, each item is represented as a sequence of discrete codes, and an autoregressive model is trained to generate the SID sequence of the next item; top-K performance is then measured by checking whether the SID sequence of the target item appears among the generated sequences. This evaluation protocol equates SID-level matching with item-level recommendation, an equivalence that holds only when every SID sequence maps to a single item. We show this assumption breaks down in practice: because tokenizers compress item features into a code space, semantically similar but collaboratively distinct items are frequently assigned the same SID sequence. Across four datasets and five representative tokenizers, the fraction of items involved in such collisions reaches 30.5%, so matching a shared SID sequence identifies only a collision group rather than the target item. Consequently, SID-level metrics overestimate item-level performance (Hit@10 is inflated by up to 103.36%), and the inflation grows with the collision rate. To support faithful comparison, we develop collision-aware item-level metrics computed directly from generated SID sequences, together with a post-tokenizer procedure that reassigns last-level SIDs at minimum cost to obtain a collision-free assignment for any existing tokenizer. Our results indicate that SID-level rankings in prior work should be interpreted with caution, and that reliable tokenizer evaluation requires either item-level correction or collision-free SID assignments.

Problem

• Semantic-ID
• generative recommendation
• tokenizer collision
• evaluation bias
• item-level recommendation

Innovation

• Semantic-ID
• collision-aware evaluation
• generative recommendation
• tokenizer bias
• item-level metrics