HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

2026-04-19

AI Summary

Existing user modeling benchmarks are largely confined to single-domain and short-term interactions, failing to address the cross-domain, long-term, and diverse modeling demands of real-world scenarios. To bridge this gap, this work proposes HORIZON—the first user modeling benchmark designed for real-world deployment—reconstructing the user modeling paradigm through dataset construction, task formulation, and evaluation protocols based on large-scale, cross-domain Amazon Reviews data. HORIZON introduces generalization challenges across domains, users, and time, defines novel tasks and evaluation metrics, and integrates sequential recommendation architectures with large language model–driven approaches for modeling long user histories, enabling pretraining and evaluation in heterogeneous environments. Experimental results reveal that current methods perform substantially worse on this benchmark, underscoring the critical need for research into temporal robustness and cross-domain generalization in user modeling.

Technology Category

• Humans and AI: User Experience and Usability
• Application Domains: Humanities & Computational Social Science
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: User model development and evaluation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

User behavior in the real world is diverse, cross-domain, and spans long time horizons. Existing user modeling benchmarks however remain narrow, focusing mainly on short sessions and next-item prediction within a single domain. Such limitations hinder progress toward robust and generalizable user models. We present HORIZON, a new benchmark that reformulates user modeling along three axes i.e. dataset, task, and evaluation. Built from a large-scale, cross-domain reformulation of Amazon Reviews, HORIZON covers 54M users and 35M items, enabling both pretraining and realistic evaluation of models in heterogeneous environments. Unlike prior benchmarks, it challenges models to generalize across domains, users, and time, moving beyond standard missing-positive prediction in the same domain. We propose new tasks and evaluation setups that better reflect real-world deployment scenarios. These include temporal generalization, sequence-length variation, and modeling unseen users, with metrics designed to assess general user behavior understanding rather than isolated next-item prediction. We benchmark popular sequential recommendation architectures alongside LLM-based baselines that leverage long-term interaction histories. Our results highlight the gap between current methods and the demands of real-world user modeling, while establishing HORIZON as a foundation for research on temporally robust, cross-domain, and general-purpose user models.

Problem

• user behavior modeling
• cross-domain
• long time horizons
• generalization
• real-world deployment

Innovation

• cross-domain user modeling
• temporal generalization
• long-term behavior
• real-world benchmark
• unseen user modeling