Trie-Aware Transformers for Generative Recommendation

2026-02-25

AI Summary

Existing generative recommendation methods, after hierarchically encoding items into token sequences, employ standard Transformers for autoregressive generation but neglect the inherent trie structure, leading to insufficient modeling of semantic relationships. This work proposes TrieRec, the first approach to integrate trie structural priors into the Transformer architecture for generative recommendation. It introduces a hyperparameter-free, model-agnostic, and efficient structure-aware positional encoding mechanism that explicitly captures item-level trie topology: an absolute positional encoding incorporating node depth and ancestor–descendant information, and a relative positional encoding that injects structural relationships into self-attention. Evaluated on four real-world datasets, TrieRec achieves an average improvement of 8.83% over strong baselines, demonstrating the effectiveness of structure-aware modeling in generative recommendation.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation
• Data Mining & Knowledge Management: Recommender Systems

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generative recommendation (GR) aligns with advances in generative AI by casting next-item prediction as token-level generation rather than score-based ranking. Most GR methods adopt a two-stage pipeline: (i) \textit{item tokenization}, which maps each item to a sequence of discrete, hierarchically organized tokens; and (ii) \textit{autoregressive generation}, which predicts the next item's tokens conditioned on the tokens of user's interaction history. Although hierarchical tokenization induces a prefix tree (trie) over items, standard autoregressive modeling with conventional Transformers often flattens item tokens into a linear stream and overlooks the underlying topology. To address this, we propose TrieRec, a trie-aware generative recommendation method that augments Transformers with structural inductive biases via two positional encodings. First, a \textit{trie-aware absolute positional encoding} aggregates a token's (node's) local structural context (\eg depth, ancestors, and descendants) into the token representation. Second, a \textit{topology-aware relative positional encoding} injects pairwise structural relations into self-attention to capture topology-induced semantic relatedness. TrieRec is also model-agnostic, efficient, and hyperparameter-free. In our experiments, we implement TrieRec within three representative GR backbones, achieving notably improvements of 8.83\% on average across four real-world datasets.

Problem

• generative recommendation
• hierarchical tokenization
• prefix tree
• structural topology
• autoregressive generation

Innovation

• trie-aware
• generative recommendation
• structural positional encoding
• hierarchical tokenization
• topology-aware attention