Not Only NTP: Extending Training Signal Coverage for Generative Recommendation

2026-07-13

AI Summary

This work addresses the limitations of Next-Token Prediction (NTP) in generative recommendation—specifically its temporal locality (restricted to single-step prediction) and spatial locality (lacking gradient pathways across domains)—by introducing two inference-free auxiliary objectives: Temporal Contrastive Learning (TCL) and Trans-Domain Learning (TDL). TCL aligns multi-step future representations via an EMA teacher model, while TDL enables cross-domain information sharing through mean-pooled hidden states with a shared prediction head, effectively capturing long-range user behavior and cross-domain patterns. Notably, this approach is the first to simultaneously overcome NTP’s training signal constraints along both temporal and spatial dimensions without adding model parameters or inference overhead. Experiments show HR@10 improvements of 34.3% on Meituan’s four-domain dataset and gains of 2.8% (HR@10) and 3.7% (NDCG@10) on Amazon benchmarks; online A/B tests further demonstrate statistically significant lifts of 1.8% in CTR and 2.1% in GMV (p<0.01).

Abstract

Next-Token Prediction (NTP) carries two structural training signal limitations. First, NTP optimizes for single-step prediction only, placing no supervised pressure on learning longer-range behavioral structure -- we term this \textbf{temporal locality}. Second, in multi-domain sequences, each target item embedding receives gradient updates exclusively from the immediately preceding hidden state, with no explicit gradient pathway from cross-domain context -- we term this \textbf{spatial locality}. We propose \textbf{NONTP}, extending NTP's signal coverage along both dimensions through two auxiliary objectives. \textbf{TCL (Temporal Contrastive Learning)} uses a BYOL-style EMA teacher with InfoNCE to align hidden states against a $K$-step future trajectory in representation space. \textbf{TDL (Trans-Domain Learning)} mean-pools cross-domain hidden states and predicts through the shared prediction head, opening a second gradient pathway with no additional parameters. Both are discarded at inference: zero overhead. On a four-domain Meituan industrial dataset (full ranking), NONTP achieves HR@10 +34.3\% over NTP and +18.3\% over MBGR. On the public Amazon Movie-Book-CDs benchmark, HR@10 +2.8\% and NDCG@10 +3.7\%. Online A/B tests confirm CTR +1.8\% and GMV +2.1\% (both $p < 0.01$). Ablation studies confirm each component contributes independently, with gradient conflict analyzed as a direction for future work.

Problem

• Next-Token Prediction
• Temporal Locality
• Spatial Locality
• Generative Recommendation
• Multi-domain Sequences

Innovation

• Temporal Contrastive Learning
• Trans-Domain Learning
• Training Signal Coverage
• Generative Recommendation
• NONTP