🤖 AI Summary
This work addresses critical limitations in Transformer-based sequential recommendation, including severe recency bias, inadequate modeling of long-term user preferences, and online-offline inconsistency caused by single-version feature storage. To tackle these issues, the authors propose the High-Inertia Long-Term Embedding (LTE) framework, which enforces cross-version consistency by anchoring content representations to fixed semantic bases. LTE embeddings are integrated as prefix tokens into a causal language model via a lagged window, effectively balancing short- and long-term interest modeling. The approach innovatively combines an asymmetric autoencoder with a lagged ensemble strategy to prevent data leakage while enhancing personalization stability. Extensive A/B testing on the Zalando platform demonstrates significant improvements in user engagement and key business metrics.
📝 Abstract
Modern transformer-based sequential recommenders excel at capturing short-term intent but often suffer from recency bias, overlooking stable long-term preferences. While extending sequence lengths is an intuitive fix, it is computationally inefficient, and recent interactions tend to dominate the model's attention. We propose Long-Term Embeddings (LTE) as a high-inertia contextual anchor to bridge this gap. We address a critical production challenge: the point-in-time consistency problem caused by infrastructure constraints, as feature stores typically host only a single "live" version of features. This leads to an offline-online mismatch during model deployments and rollbacks, as models are forced to process evolved representations they never saw during training. To resolve this, we introduce an LTE framework that constrains embeddings to a fixed semantic basis of content-based item representations, ensuring cross-version compatibility. Furthermore, we investigate integration strategies for causal language modeling, considering the data leakage issue that occurs when the LTE and the transformer's short-term sequence share a temporal horizon. We evaluate two representations: a heuristic average and an asymmetric autoencoder with a fixed decoder grounded in the semantic basis to enable behavioral fine-tuning while maintaining stability. Online A/B tests on Zalando demonstrate that integrating LTE as a contextual prefix token using a lagged window yields significant uplifts in both user engagement and financial metrics.