Rethinking Item Tokenization in Generative Recommenders: From Fixed Atoms to Semantic Subwords

2026-08-23

AI Summary

本文提出语义子词分词方法，解决生成推荐系统中因细粒度表示导致的内部项目注意力过载问题。

Abstract

In generative recommender systems, items are typically tokenized into fixed-length semantic ID sequences for autoregressive next-item prediction. However, for user-context modeling, this fine-grained representation triggers Intra-item Attention Overload: excessive attention is spent on low-level intra-item dependencies rather than high-level inter-item behavioral transitions. To address this, we propose Semantic Subword Tokenization (SST), which represents historical items as variable-length semantic subwords while preserving fixed-length target decoding. SST first applies Item-level Subword Tokenization (IST) to merge stable adjacent atom tokens into compact semantic subword tokens, thereby reducing intra-item reassembly in the encoder. It then introduces Behavior-induced Co-occurrence Augmentation (BCA) to inject coarse-grained semantic prefix transition signals, guiding the freed modeling capacity toward inter-item behavioral regularities. Extensive experiments on three public datasets and three generative recommender backbones show empirical improvements of SST over fixed-length and transferable variable-length SID baselines. Code is available at https://github.com/mxrcandy/Semantic-Subword-Tokenization.

Problem

• Intra-item Attention Overload
• Generative Recommender Systems
• Semantic ID Sequences

Innovation

• Semantic Subword Tokenization
• Item-level Subword Tokenization
• Behavior-induced Co-occurrence Augmentation