HorizonBench: Long-Horizon Personalization with Evolving Preferences

2026-04-19

AI Summary

This work addresses the challenge of modeling dynamic user preferences shaped by life events, a problem hindered by the absence of long-term conversational data with grounded annotations of preference shifts. We introduce the first benchmark dataset, HorizonBench, explicitly designed for long-horizon personalized modeling, featuring authentic rationales for preference evolution. Built upon a structured mental-state graph-driven data generator, HorizonBench encompasses 360 simulated users, 4,245 tasks, and an average of 4,300 dialogue turns per user. Comprehensive evaluation of 25 state-of-the-art models reveals that even the best achieves only 52.8% accuracy in tracking user state changes, underscoring the task’s difficulty and highlighting the benchmark’s potential to advance research in memory-augmented systems and theory-of-mind reasoning.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

User preferences evolve across months of interaction, and tracking them requires inferring when a stated preference has been changed by a subsequent life event. We define this problem as long-horizon personalization and observe that progress on it is limited by data availability and measurement, with no existing resource providing both naturalistic long-horizon interactions and the ground-truth provenance needed to diagnose why models fail. We introduce a data generator that produces conversations from a structured mental state graph, yielding ground-truth provenance for every preference change across 6-month timelines, and from it construct HorizonBench, a benchmark of 4,245 items from 360 simulated users with 6-month conversation histories averaging ~4,300 turns and ~163K tokens. HorizonBench provides a testbed for long-context modeling, memory-augmented architectures, theory-of-mind reasoning, and user modeling. Across 25 frontier models, the best model reaches 52.8% and most score at or below the 20% chance baseline. When these models err on evolved preferences, over a third of the time they select the user's originally stated value without tracking the updated user state. This belief-update failure persists across context lengths and expression explicitness levels, identifying state-tracking capability as the primary bottleneck for long-horizon personalization.

Problem

• long-horizon personalization
• evolving preferences
• preference tracking
• user modeling
• state tracking

Innovation

• long-horizon personalization
• preference evolution
• mental state graph
• ground-truth provenance
• state-tracking