Addressing Personalized Bias for Unbiased Learning to Rank

2025-08-28

AI Summary

Existing unbiased learning to rank (ULTR) methods assume user behavior logs originate from a homogeneous “average user,” ignoring heterogeneity in query intent and document browsing preferences across individuals—introducing bias. Method: This paper formally introduces the user-aware ULTR problem, modeling personalized biases in user–query–document ternary interactions. Leveraging causal inference, we propose a low-variance user-aware inverse propensity scoring (IPS) estimator that jointly models user behavior distributions and aggregates query-level weights. Contribution/Results: Evaluated on two semi-synthetic and one real-world dataset, our method significantly outperforms state-of-the-art ULTR baselines in ranking accuracy while improving fairness. It establishes a novel paradigm for unbiased learning under personalization, bridging the gap between traditional ULTR and user-specific behavioral modeling.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Unbiased learning to rank (ULTR), which aims to learn unbiased ranking models from biased user behavior logs, plays an important role in Web search. Previous research on ULTR has studied a variety of biases in users' clicks, such as position bias, presentation bias, and outlier bias. However, existing work often assumes that the behavior logs are collected from an ``average'' user, neglecting the differences between different users in their search and browsing behaviors. In this paper, we introduce personalized factors into the ULTR framework, which we term the user-aware ULTR problem. Through a formal causal analysis of this problem, we demonstrate that existing user-oblivious methods are biased when different users have different preferences over queries and personalized propensities of examining documents. To address such a personalized bias, we propose a novel user-aware inverse-propensity-score estimator for learning-to-rank objectives. Specifically, our approach models the distribution of user browsing behaviors for each query and aggregates user-weighted examination probabilities to determine propensities. We theoretically prove that the user-aware estimator is unbiased under some mild assumptions and shows lower variance compared to the straightforward way of calculating a user-dependent propensity for each impression. Finally, we empirically verify the effectiveness of our user-aware estimator by conducting extensive experiments on two semi-synthetic datasets and a real-world dataset.

Problem

• Addressing personalized bias in unbiased learning to rank
• Modeling user-specific browsing behaviors for each query
• Developing user-aware inverse-propensity-score estimator for ranking

Innovation

• User-aware inverse-propensity-score estimator for ranking
• Models user browsing distributions per query
• Aggregates user-weighted examination probabilities