Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation

2026-08-07

AI Summary

This work addresses a critical limitation in existing large language model–based generative recommender systems, where randomly initialized semantic IDs disrupt the geometric structure of the original embedding space, leading to overreliance on item popularity and degraded generalization and cold-start performance. To mitigate this, the authors propose a plug-and-play initialization strategy that directly sets each semantic ID’s embedding to the centroid of its corresponding semantic cluster in the original embedding space. This approach preserves semantic geometry without introducing additional parameters, training overhead, inference cost, or requiring continual pretraining. Experimental results demonstrate that, with only supervised fine-tuning (SFT), the method improves Recall@5 by up to 16% and accelerates convergence by 40%. In cold-start scenarios, Recall@5 gains reach 60%, and when combined with continual pretraining, it achieves comparable performance in half the training epochs.

Abstract

Recent advances in generative recommendation (GR) leverage large language models (LLMs) as recommender backbones, enabling LLMs to directly generate recommendations conditioned on item-interaction histories. In these systems, items are often represented through semantic IDs (SIDs) added to the LLM vocabulary as special tokens. Ideally, SIDs imbue item token representations with semantic priors, thereby improving model generalization. However, standard vocabulary expansion typically initializes these tokens as random Gaussian vectors, discarding the SIDs' underlying continuous geometry and forcing the LLM to relearn token relationships from interaction data. To demonstrate the consequences of this design, we first show that training from this initialization tends to organize SID embeddings around item popularity rather than semantics. We further show that, despite partially reducing the reliance on popularity and improving cold item performance, the computationally expensive process of continual pretraining (CPT) fails to reliably recover the original semantic geometry. To address these findings, we propose a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space. Requiring only a few lines of code and no additional training or inference overhead, this drop-in approach improves pure-SFT Recall@5 by up to 16%, reaches peak performance with up to 40% fewer SFT steps, and improves cold-item Recall@5 by up to 60%. Moreover, on datasets that benefit from additional CPT, centroid initialization reaches comparable performance while requiring half as many CPT epochs. Together, our findings show that preserving SID geometry, beyond shared-prefix structure, provides a simple and effective semantic prior for LLM-based GR.

Problem

• generative recommendation
• semantic ID
• token initialization
• LLM-based recommendation
• embedding geometry

Innovation

• semantic ID initialization
• centroid embedding
• generative recommendation
• LLM-based recommendation
• cold-start performance